Image Captioning Trials¶

In [ ]:
%pip install -q -r requirements.txt
Note: you may need to restart the kernel to use updated packages.
In [ ]:
import tensorflow as tf
from tensorflow.keras import layers, regularizers
from tensorflow.keras.models import Sequential, Model
from onedrivedownloader import download
import os
import shutil
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import PIL
import datetime
import requests
from tensorboard.plugins.hparams import api as hp
import visualkeras
import zipfile
import json
from tqdm import tqdm
import collections
import nltk
from nltk.translate.bleu_score import corpus_bleu
import time

# Make sure to download NLTK data for BLEU if not already installed
nltk.download('punkt')
2024-10-24 09:57:19.196963: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-10-24 09:57:19.210664: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-10-24 09:57:19.214395: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-24 09:57:19.224620: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-24 09:57:19.966669: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[nltk_data] Downloading package punkt to /home/arslane/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
True
In [ ]:
os.environ['CUDA_VISIBLE_DEVICES'] = "0"

gpus = tf.config.experimental.list_physical_devices('GPU')
print(gpus)
if gpus:
    try:
        # Enable memory growth for each GPU
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Memory growth must be set before GPUs have been initialized
        print(e)
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1729756643.293251    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.332427    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.334481    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.337547    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.339208    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.340815    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.460215    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.462108    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1729756643.463745    9183 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-10-24 09:57:23.465325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2179 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2050, pci bus id: 0000:01:00.0, compute capability: 8.6
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
1 Physical GPUs, 1 Logical GPUs

Loading the COCO dataset¶

In [ ]:
# URLs for the zip files
annotations_url = "https://viacesifr-my.sharepoint.com/:u:/r/personal/salma_turki_viacesi_fr/Documents/COCO%20Dataset/annotations_trainval2014.zip?csf=1&web=1&e=JJ2vXX&download=1"
train2014_url = "https://viacesifr-my.sharepoint.com/:u:/r/personal/salma_turki_viacesi_fr/Documents/COCO%20Dataset/train2014.zip?csf=1&web=1&e=e4CdT1&download=1"

# Local paths
annotation_folder = os.path.join(".", "annotations")
train2014_folder = os.path.join(".", "train2014")
annotation_zip = os.path.join(".", "annotations.zip")
train2014_zip = os.path.join(".", "train2014.zip")

# Check if the folders already exist
if not os.path.exists(annotation_folder):
    print("annotations folder not found. Downloading...")
    # Download the annotations.zip file
    response = requests.get(annotations_url, stream=True)
    with open(annotation_zip, 'wb') as f:
        shutil.copyfileobj(response.raw, f)
    del response

    # Unzip the annotations.zip
    print("Extracting annotations.zip...")
    with zipfile.ZipFile(annotation_zip, 'r') as zip_ref:
        zip_ref.extractall(annotation_folder)

    # Delete the annotations.zip file
    print("Deleting annotations.zip...")
    os.remove(annotation_zip)

if not os.path.exists(train2014_folder):
    print("train2014 folder not found. Downloading...")
    # Download the train2014.zip file
    response = requests.get(train2014_url, stream=True)
    with open(train2014_zip, 'wb') as f:
        shutil.copyfileobj(response.raw, f)
    del response

    # Unzip the train2014.zip
    print("Extracting train2014.zip...")
    with zipfile.ZipFile(train2014_zip, 'r') as zip_ref:
        zip_ref.extractall(train2014_folder)

    # Delete the train2014.zip file
    print("Deleting train2014.zip...")
    os.remove(train2014_zip)

print("Download and extraction completed.")
Download and extraction completed.
In [ ]:
def get_feature_extraction_model(model_choice='InceptionV3'):
    """
    Returns the feature extraction model, preprocessing function, image size, and feature shapes based on the chosen model.
    """
    if model_choice == 'InceptionV3':
        image_model = tf.keras.applications.InceptionV3(include_top=False, weights='imagenet')
        preprocess_input = tf.keras.applications.inception_v3.preprocess_input
        img_size = (299, 299)  # Image size required for InceptionV3
        attention_features_shape = 64  # 8 * 8
        features_shape = 2048  # Depth of the feature map
    elif model_choice == 'ResNet50':
        image_model = tf.keras.applications.ResNet50(include_top=False, weights='imagenet')
        preprocess_input = tf.keras.applications.resnet50.preprocess_input
        img_size = (224, 224)  # Image size required for ResNet50
        attention_features_shape = 49  # 7 * 7
        features_shape = 2048  # Depth of the feature map
    else:
        raise ValueError("model_choice must be either 'InceptionV3' or 'ResNet50'")

    new_input = image_model.input
    hidden_layer = image_model.layers[-1].output
    image_features_extract_model = tf.keras.Model(new_input, hidden_layer)

    return image_features_extract_model, preprocess_input, img_size, features_shape, attention_features_shape
In [ ]:
model_choice = 'InceptionV3'  # Change this to 'ResNet50' to use ResNet
image_features_extract_model, preprocess_input, img_size, features_shape, attention_features_shape = get_feature_extraction_model(model_choice)
In [ ]:
# model_choice = 'ResNet50'  # Change this to 'InceptionV3' to use ResNet
# image_features_extract_model, preprocess_input, img_size, features_shape, attention_features_shape = get_feature_extraction_model(model_choice)
In [ ]:
# Annotation file path
annotation_file = os.path.join(annotation_folder, "captions_train2014.json")

# Lecture du fichier d'annotation
with open(annotation_file, 'r') as f:
    annotations = json.load(f)

# Group all annotations with the same identifier.
image_path_to_caption = collections.defaultdict(list)
for val in annotations['annotations']:
    # mark the beginning and end of each annotation
    caption = f"<start> {val['caption']} <end>"
    # An image's identifier is part of its access path.
    image_path = os.path.join(train2014_folder, 'COCO_train2014_' + '%012d.jpg' % (val['image_id']))
    # Add caption to image_path
    image_path_to_caption[image_path].append(caption)
    
# Take first images only
image_paths = list(image_path_to_caption.keys())
train_image_paths = image_paths[:5000]

# List of all annotations
train_captions = []
# List of all duplicated image file names (in number of annotations per image)
img_name_vector = []

for image_path in train_image_paths:
    caption_list = image_path_to_caption[image_path]
    # Add caption_list to train_captions
    train_captions.extend(caption_list)
    # Add duplicate image_path len(caption_list) times
    img_name_vector.extend([image_path] * len(caption_list))

print(f"Number of images: {len(train_image_paths)}")
Number of images: 5000
In [ ]:
# Function to load and preprocess image
def load_image(image_path):
    """
    Load and preprocess the image according to the model (InceptionV3 or ResNet50)
    The load_image function has as input the path of an image and as output a pair
    pair containing the processed image and its path.
    The load_image function performs the following processing:
        1. Loads the file corresponding to the path image_path
        2. Decodes the image into RGB.
        3. Resize image.
        4. Normalize image pixels between -1 and 1.
    """
    img = tf.io.read_file(image_path)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.resize(img, img_size)  # Adjust image size based on model
    img = preprocess_input(img)  # Preprocess input according to selected model
    return img, image_path

# Image preprocessing for the dataset
encode_train = sorted(set(img_name_vector))

image_dataset = tf.data.Dataset.from_tensor_slices(encode_train)
image_dataset = image_dataset.map(load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE).batch(8)

# Batch processing for feature extraction
for img, path in tqdm(image_dataset):
    batch_features = image_features_extract_model(img)
    batch_features = tf.reshape(batch_features, (batch_features.shape[0], -1, batch_features.shape[3]))
    for bf, p in zip(batch_features, path):
        path_of_feature = p.numpy().decode("utf-8")
        np.save(path_of_feature, bf.numpy())

# Display batch features shape
for img, path in image_dataset:
    batch_features = image_features_extract_model(img)
    print(f"Batch features shape: {batch_features.shape}")
    break
  0%|          | 0/625 [00:00<?, ?it/s]2024-10-24 09:57:26.536374: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 8907
W0000 00:00:1729756646.591262    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.606570    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.608286    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.615191    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.617683    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.619977    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.622065    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.624676    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.626764    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.665406    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.678616    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.681239    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.684857    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.688107    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.691618    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.695449    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.699599    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.702966    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.707117    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.710340    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.713572    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.718296    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.721848    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.726088    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.748091    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.752708    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.757967    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.762400    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.767308    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.776802    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.896441    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.898945    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.901453    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.904448    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.907631    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.910673    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.913774    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.916736    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.920137    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.923969    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.927797    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.931085    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.934055    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.937309    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.941340    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.944640    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.948244    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.952392    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.956594    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.960846    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.990076    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.994263    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756646.999324    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.003095    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.006804    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.010999    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.015235    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.018717    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.023306    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.027882    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.032475    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.037224    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.042793    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.049158    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.055670    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.059968    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.064308    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.081042    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.105462    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.108332    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.111248    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.114200    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.117123    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.120065    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.123665    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.127972    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.131582    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.135459    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.139388    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.143184    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.147132    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.151014    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.154964    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.157875    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.161461    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.165382    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.169866    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.174670    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.179669    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.184566    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.189118    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.200489    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.205915    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.211935    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.217560    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.223027    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.229334    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.236608    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.240464    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.244487    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.248274    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.256531    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.261488    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.273098    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
2024-10-24 09:57:27.279905: W external/local_tsl/tsl/framework/bfc_allocator.cc:363] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
W0000 00:00:1729756647.307655    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.342451    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.393213    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.396629    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.400391    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.404291    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.408090    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.411815    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.415585    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.419905    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.424291    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.428638    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.432360    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.436927    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.441330    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.445563    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.450110    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.453784    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.458480    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.463494    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.467997    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.472481    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.476093    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.480912    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.485254    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.490550    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.494932    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.501023    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.507152    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.511328    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.515681    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.520856    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.526015    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.531554    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.537258    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.542958    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.548222    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.553837    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.560532    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.567254    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.592521    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.594228    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.595886    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.597521    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.599137    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.600714    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.602320    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.604029    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.605590    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.607172    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.608733    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.610270    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.611881    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.613463    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.615095    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.616769    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.618367    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.620130    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.621728    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.623375    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.625079    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.626719    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.628386    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.630095    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.631805    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.633469    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.635210    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.637007    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.638739    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.640448    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.642327    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.644260    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.645977    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.647689    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.649376    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.651383    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.653270    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.655255    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.657114    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.659007    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.699986    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.701793    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.703533    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.705253    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.706966    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.708710    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.710431    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.712220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.713927    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.715790    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.717590    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.719317    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.721187    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.722901    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.724600    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.726356    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.728203    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.730031    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.731959    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.733830    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.735799    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.737623    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.739539    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.741406    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.743315    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.745173    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.747154    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.749073    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.750971    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.752892    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.754798    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.756630    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.758597    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.760555    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.762964    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.765156    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.767352    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.769327    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.771594    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.773805    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.789430    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.791087    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.792873    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.794687    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.796434    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.798181    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.799861    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.801580    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.803295    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.805144    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.806952    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.808833    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.810504    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.812247    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.814089    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.815891    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.817699    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.819507    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.821479    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.823397    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.825399    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.827464    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.829440    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.831428    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.833479    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.835283    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.837075    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.838832    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.841053    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.843223    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.845185    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.848005    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.850419    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.936180    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.937837    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.939343    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.940873    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.942459    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.943961    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.945486    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.947026    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.948596    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.950158    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.951723    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.953284    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.954786    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.956311    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.957822    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.959331    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.960912    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.962431    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.963934    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.965436    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.966977    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.968508    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.970051    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.971766    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.973375    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.974878    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.976405    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.977955    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.979538    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.981107    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.982695    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.984262    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.985879    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.987485    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.989039    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.990595    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.992316    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.993993    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.995587    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756647.997153    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.012694    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.015153    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.017746    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.020177    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.022719    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.025256    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.027685    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.030390    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.032802    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.036165    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.038799    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.041632    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.043978    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.046288    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.048841    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.051381    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.053860    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.056362    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.059208    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.061655    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.064147    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.066634    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.069553    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.072284    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.074837    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.077563    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.080264    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.083283    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.086247    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.089542    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.092484    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.095777    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.098842    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.102144    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.105845    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.109563    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.112810    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.116944    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.127657    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.156086    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.157600    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.159033    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.160482    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.161990    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.163509    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.165030    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.166512    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.167971    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.169399    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.170854    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.172283    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.173724    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.175177    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.176653    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.178147    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.179621    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.181082    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.182551    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.184036    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.185506    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.187016    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.188620    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.190087    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.191548    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.193038    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.194556    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.200486    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.202124    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.203778    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.205353    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.206850    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.208339    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.209907    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.211720    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.213585    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.215169    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.216738    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.218400    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.221428    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.245066    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.246530    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.247954    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.249408    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.250888    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.252369    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.253820    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.255260    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.256699    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.258173    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.259635    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.261099    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.262531    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.263962    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.265392    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.266861    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.268322    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.269802    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.271266    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.272758    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.274254    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.275751    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.277202    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.278688    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.280188    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.281907    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.283480    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.285009    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.286515    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.288128    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.289703    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.291189    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.292813    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.294639    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.296218    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.297769    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.299431    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.300996    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.302835    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.305816    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.335300    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.336808    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.338293    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.339824    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.341309    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.342821    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.344367    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.345890    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.347385    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.348896    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.350434    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.351903    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.353452    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.354940    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.356472    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.358076    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.359571    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.361111    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.362699    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.364311    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.365954    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.367463    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.369056    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.370690    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.372216    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.373773    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.375403    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.377068    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.378715    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.380373    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.381994    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.383614    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.385359    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.387019    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.389060    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.398219    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.400220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.401981    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.403761    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.407530    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.433178    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.434688    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.436166    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.437636    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.439181    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.440679    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.442220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.443767    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.445279    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.446755    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.448335    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.449849    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.451329    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.452823    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.454321    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.455824    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.457382    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.458972    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.460472    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.462098    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.463636    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.465298    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.466931    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.468459    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.470062    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.471721    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.473244    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.474797    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.476420    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.478051    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.479682    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.481310    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.482967    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.484662    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.486429    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.488176    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.489954    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.493749    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.495769    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.497936    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.532528    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.534076    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.535598    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.537094    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.538648    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.540213    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.541789    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.543386    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.544984    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.546592    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.548100    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.549647    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.551274    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.552805    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.554445    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.556134    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.557656    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.559256    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.560888    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.562396    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.563989    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.565722    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.567308    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.568870    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.570470    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.572147    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.573821    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.575545    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.577161    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.578889    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.580577    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.582262    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.584021    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.585770    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.587549    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.589592    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.591545    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.593510    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.607450    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.609006    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.610522    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.612022    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.613587    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.615172    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.616772    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.618320    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.619920    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.621461    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.623059    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.624722    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.626357    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.627871    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.629564    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.631071    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.632713    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.634298    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.635888    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.637432    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.638937    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.640533    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.642161    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.643894    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.645460    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.647117    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.648852    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.650515    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.652135    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.653775    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.655508    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.657174    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.658928    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.660702    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.662744    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.664479    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.666534    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.668408    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.670289    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.674101    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.708437    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.709989    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.711504    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.713008    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.714578    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.716160    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.717767    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.719320    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.720917    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.722456    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.724060    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.725775    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.727413    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.728962    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.730686    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.732236    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.733834    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.735414    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.737028    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.738570    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.740128    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.741772    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.743425    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.745151    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.746758    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.748474    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.750198    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.751911    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.753610    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.755298    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.757030    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.758725    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.760552    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.762364    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.764590    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.766339    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.768513    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.770379    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.772273    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.776763    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.807125    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.808696    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.810223    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.811725    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.813285    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.814853    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.816421    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.818021    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.819614    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.821213    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.822754    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.824295    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.825926    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.827459    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.829101    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.830787    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.832340    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.833939    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.835665    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.837209    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.838802    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.840530    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.842122    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.843726    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.845356    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.847090    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.848777    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.850506    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.852214    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.853937    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.855634    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.857341    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.859179    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.860917    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.862678    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.864872    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.866825    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.868794    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.905072    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.906672    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.908241    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.909797    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.911438    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.913028    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.914580    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.916181    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.917796    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.919548    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.921161    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.922829    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.924502    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.926267    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.927989    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.929737    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.931447    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.933063    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.934713    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.936366    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.938090    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.939793    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.941506    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.943089    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.944917    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.946586    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.948380    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.950225    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.952117    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.953922    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.955719    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.957690    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.959622    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.961449    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.963340    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.965757    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.967908    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.969976    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.972360    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756648.977506    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.023280    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.024891    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.026497    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.028136    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.029744    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.031367    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.033047    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.034906    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.036510    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.038157    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.039880    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.041492    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.043238    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.044937    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.046516    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.048198    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.050060    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.051741    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.053417    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.055089    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.056841    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.058615    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.060292    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.061936    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.063608    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.065286    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.067126    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.068926    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.070687    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.072563    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.074380    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.076160    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.078170    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.080055    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.081991    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.084454    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.086496    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.088905    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.090942    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.095963    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.154374    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.155871    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.157313    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.158755    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.160169    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.161615    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.163044    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.164481    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.166567    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.168035    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.169535    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.171005    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.172417    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.173832    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.175290    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.176713    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.178139    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.179599    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.181149    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.182616    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.184177    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.185620    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.187072    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.188556    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.190057    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.191538    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.193020    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.194491    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.195993    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.197514    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.198975    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.200494    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.202011    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.203579    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.205118    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.206717    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.208319    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.209814    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.211463    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.213085    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.225407    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.226913    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.228395    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.229944    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.231450    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.232987    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.234564    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.236083    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.237619    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.239179    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.240687    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.242186    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.243705    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.245255    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.246783    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.248311    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.249813    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.251406    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.253030    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.254535    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.256051    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.257567    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.259106    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.260617    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.262118    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.263631    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.265142    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.266787    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.268436    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.270087    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.271898    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.273731    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.275271    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.276971    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.307228    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.308921    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.310593    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.312253    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.313929    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.315699    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.317511    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.319267    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.321020    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.322762    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.324489    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.326224    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.328026    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.329788    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.331594    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.333391    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.335244    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.337145    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.339044    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.340803    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.342529    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.344255    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.346029    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.347982    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.349730    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.351474    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.353236    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.355005    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.356767    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.358543    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.360814    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.362995    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.364734    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.367124    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.369173    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.371115    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.373178    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.375186    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.377846    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.397772    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.399279    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.400750    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.402164    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.403589    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.405061    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.406545    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.407987    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.409499    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.410975    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.412474    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.413955    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.415422    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.416912    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.418371    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.419899    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.421385    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.422840    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.424336    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.425793    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.427319    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.428911    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.430356    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.431788    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.433239    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.434718    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.436180    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.437613    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.439141    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.440830    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.442339    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.443859    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.445400    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.447097    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.448635    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.450319    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.451927    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.465625    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.467102    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.468556    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.469997    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.471435    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.472898    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.474367    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.475887    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.477361    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.478797    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.480242    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.481741    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.483225    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.484682    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.486167    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.487717    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.489261    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.490711    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.492141    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.493571    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.495092    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.496536    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.497981    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.499425    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.500880    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.502320    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.503796    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.505217    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.506864    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.508409    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.509924    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.511458    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.512962    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.514635    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729756649.516150    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
100%|██████████| 625/625 [01:56<00:00,  5.08it/s]2024-10-24 09:59:23.203779: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
100%|██████████| 625/625 [01:56<00:00,  5.35it/s]
Batch features shape: (8, 8, 8, 2048)

Pre-processing¶

Annotation pre-processing¶

In [ ]:
# Find the maximum size
def calc_max_length(tensor):
    return max(len(t) for t in tensor)

# Choose the 5000 most frequent words in the vocabulary
top_k = 5000
# The Tokenizer class enables text pre-processing for neural networks
tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=top_k,
                                                  oov_token="<unk>",
                                                  filters='!"#$%&()*+.,-/:;=?@[\]^_`{|}~ ')
# Builds a vocabulary based on the train_captions list
tokenizer.fit_on_texts(train_captions)

# Create the token used to fill annotations to equalize their length
tokenizer.word_index['<pad>'] = 0
tokenizer.index_word[0] = '<pad>'

# Creation of vectors (list of integer tokens) from annotations (list of words)
train_seqs = tokenizer.texts_to_sequences(train_captions)

# Fill each vector up to the maximum annotation length
cap_vector = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')

# Calculates the maximum length used to store attention weights
# It will later be used for display during evaluation.
max_length = calc_max_length(train_seqs)

Formation of a training and test set¶

In [ ]:
img_to_cap_vector = collections.defaultdict(list)
# Creation of a dictionary associating image paths (.npy file) with annotations 
# # Images are duplicated because there are several annotations per image
print(len(img_name_vector), len(cap_vector))
for img, cap in zip(img_name_vector, cap_vector):
    img_to_cap_vector[img].append(cap)

"""
Creation of training and validation datasets 
using a random 80-20 split
""" 
# Take the keys (names of processed image files), *these will not be duplicated*.
img_keys = list(img_to_cap_vector.keys())
# Dividing clues into training and testing
slice_index = int(len(img_keys)*0.8)
img_name_train_keys, img_name_val_keys = img_keys[:slice_index], img_keys[slice_index:]

"""
Training and test games are in the form of
lists containing mappings:(pre-processed image ---> annotation token(word) )
"""

# Loop to build the training set
img_name_train = []
cap_train = []
for imgt in img_name_train_keys:
    capt_len = len(img_to_cap_vector[imgt])
    # Duplication of images by number of annotations per image
    img_name_train.extend([imgt] * capt_len)
    cap_train.extend(img_to_cap_vector[imgt])

# Loop to build the test set
img_name_val = []
cap_val = []
for imgv in img_name_val_keys:
    capv_len = len(img_to_cap_vector[imgv])
    # Duplication of images by number of annotations per image
    img_name_val.extend([imgv] * capv_len)
    cap_val.extend(img_to_cap_vector[imgv])

len(img_name_train), len(cap_train), len(img_name_val), len(cap_val)
25011 25011
(20008, 20008, 5003, 5003)
In [ ]:
BATCH_SIZE = 16 # batch size
BUFFER_SIZE = 1000 # buffer size for data mixing
embedding_dim = 256
units = 512 # Hidden layer size in RNN
vocab_size = top_k + 1
num_steps = len(img_name_train) // BATCH_SIZE

# Function that loads numpy files from pre-processed images
def map_func(img_name, cap):
    img_tensor = np.load(img_name.decode('utf-8')+'.npy')
    return img_tensor, cap

# Creation of a "Tensor "s dataset (used to represent large datasets)
# The dataset is created from "img_name_train" and "cap_train".
dataset = tf.data.Dataset.from_tensor_slices((img_name_train, cap_train))

# Use map to load numpy files (possibly in parallel)
dataset = dataset.map(lambda item1, item2: tf.numpy_function(
          map_func, [item1, item2], [tf.float32, tf.int32]),
          num_parallel_calls=tf.data.experimental.AUTOTUNE)

# Mixing data and dividing them into batches
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
In [ ]:
import tensorflow as tf
from abc import ABC, abstractmethod

# Base Encoder Class
class BaseEncoder(tf.keras.Model, ABC):
    @abstractmethod
    def call(self, x):
        pass

# Base Decoder Class
class BaseDecoder(tf.keras.Model, ABC):
    @abstractmethod
    def call(self, x, features, hidden):
        pass

CNN¶

In [ ]:
class CNN_Encoder(BaseEncoder):
    def __init__(self, embedding_dim):
        super(CNN_Encoder, self).__init__()
        self.fc = tf.keras.layers.Dense(embedding_dim)

    def call(self, x):
        x = self.fc(x)
        x = tf.nn.relu(x)
        return x

Attention mechanisms¶

Bahdanau¶

In [ ]:
class BahdanauAttention(tf.keras.Model):
    def __init__(self, units):
        super(BahdanauAttention, self).__init__()
        # W1 is the hidden layer for image features
        self.W1 = tf.keras.layers.Dense(units)
        # W2 is the hidden layer for the previous hidden layer
        self.W2 = tf.keras.layers.Dense(units)
        # V is the output layer that gives a non-normalized score for each image feature
        self.V = tf.keras.layers.Dense(1)

    def call(self, features, hidden):
        hidden_with_time_axis = tf.expand_dims(hidden, 1)

        attention_hidden_layer = (tf.nn.tanh(self.W1(features) + self.W2(hidden_with_time_axis)))

        # This gives you a non-normalized score for each image feature.
        score = self.V(attention_hidden_layer)

        attention_weights = tf.nn.softmax(score, axis=1)

        context_vector = attention_weights * features
        context_vector = tf.reduce_sum(context_vector, axis=1)
        
        return context_vector, attention_weights

RNN¶

GRU¶

1 layer¶

In [ ]:
class RNN_Decoder_GRU(BaseDecoder):
    def __init__(self, embedding_dim, units, vocab_size):
        super(RNN_Decoder_GRU, self).__init__()
        self.units = units
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        # self.gru = tf.keras.layers.GRU(self.units,
        #                                return_sequences=True,
        #                                return_state=True,
        #                                recurrent_initializer='glorot_uniform')
        self.gru = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
        self.fc1 = tf.keras.layers.Dense(self.units)
        self.fc2 = tf.keras.layers.Dense(vocab_size)

        self.attention = BahdanauAttention(self.units)

    def call(self, x, features, hidden):
        context_vector, attention_weights = self.attention(features, hidden)
        
        # Ensure x has a time dimension
        x = self.embedding(x)
        if len(x.shape) == 2:  # if shape is [batch_size, embedding_dim]
            x = tf.expand_dims(x, 1)  # expand dims to [batch_size, 1, embedding_dim]
        
        # Concatenate context vector with x
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

        # Pass the concatenated tensor to the GRU
        output, state = self.gru(x)

        y = self.fc1(output)
        y = tf.reshape(y, (-1, y.shape[2]))  # Flatten before final Dense layer
        y = self.fc2(y)

        return y, state, attention_weights


    def reset_state(self, batch_size):
        return tf.zeros((batch_size, self.units))

3 layers¶

In [ ]:
class RNN_Decoder_GRU_3L(tf.keras.Model):
    def __init__(self, embedding_dim, units, vocab_size):
        super(RNN_Decoder_GRU_3L, self).__init__()
        self.units = units

        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        # self.gru = tf.keras.layers.GRU(self.units, # Taille de la couche cachée du GRU
        #                                return_sequences=True, # retourne la séquence complète de sortie de chaque pas de temps
        #                                return_state=True, # retourne l'état caché de la dernière étape de temps
        #                                recurrent_initializer='glorot_uniform') # glorot_uniform est une initialisation des poids qui permet de mieux converger lors de l'entrainement en utilisant la fonction d'activation relu ce qui fait que les poids sont initialisés de manière à ce que la variance de la sortie soit égale à la variance de l'entrée
        # 3 couches de GRU
        self.gru1 = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
        self.gru2 = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
        self.gru3 = tf.keras.layers.RNN(tf.keras.layers.GRUCell(self.units), return_sequences=True, return_state=True)
        #Couche dense qui aura pour entrée la sortie du GRU
        self.fc1 = tf.keras.layers.Dense(self.units)
        # Dernière couche dense
        self.fc2 = tf.keras.layers.Dense(vocab_size)

        self.attention = BahdanauAttention(self.units)

    def call(self, x, features, hidden):
        # L'attention est defini par un modèle a part
        context_vector, attention_weights = self.attention(features, hidden)
        # Passage du mot courant à la couche embedding
        x = self.embedding(x)
        # Concaténation
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1) #DONE tf.expand_dims permet de rajouter une dimension à un tenseur à une position donnée

        # Passage du vecteur concaténé à la gru
        output, state = self.gru1(x)
        output, state = self.gru2(output)
        output, state = self.gru3(output)
        
        # Couche dense
        y = self.fc1(output)

        y = tf.reshape(y, (-1, x.shape[2])) # Aplatir le tenseur pour le passer à la couche dense suivante (fc2)
        
        # Couche dense
        y = self.fc2(y)
        
        return y, state, attention_weights

    def reset_state(self, batch_size):
        return tf.zeros((batch_size, self.units))

LSTM¶

In [ ]:
class RNN_Decoder_LSTM(BaseDecoder):
    def __init__(self, embedding_dim, units, vocab_size):
        super(RNN_Decoder_LSTM, self).__init__()
        self.units = units
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        self.lstm = tf.keras.layers.LSTM(self.units, return_sequences=True, return_state=True)
        self.fc1 = tf.keras.layers.Dense(self.units)
        self.fc2 = tf.keras.layers.Dense(vocab_size)
        self.attention = BahdanauAttention(self.units)

    def call(self, x, features, hidden):
        context_vector, attention_weights = self.attention(features, hidden)
        x = self.embedding(x)
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
        output, hidden_state, cell_state = self.lstm(x)
        print("OUTPUT:", output.shape)
        y = self.fc1(output)
        y = tf.reshape(y, (-1, x.shape[2]))
        y = self.fc2(y)
        return y, hidden_state, attention_weights
    
    def reset_state(self, batch_size):
        return tf.zeros((batch_size, self.units))

Bidirectional LSTM¶

In [ ]:
class RNN_Decoder_Bidirectional_LSTM(BaseDecoder):
    def __init__(self, embedding_dim, units, vocab_size):
        super(RNN_Decoder_Bidirectional_LSTM, self).__init__()
        self.units = units
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        self.bilstm = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(self.units, return_sequences=True, return_state=True))
        # self.lstm = tf.keras.layers.LSTM(self.units, return_sequences=True, return_state=True)
        self.fc0 = tf.keras.layers.Dense(self.units)
        self.fc1 = tf.keras.layers.Dense(self.units)
        self.fc2 = tf.keras.layers.Dense(vocab_size)
        self.attention = BahdanauAttention(self.units)
        self.fc_hidden = tf.keras.layers.Dense(self.units)
        
    def call(self, x, features, hidden):
        context_vector, attention_weights = self.attention(features, hidden)
        x = self.embedding(x)
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
        bi_output, forward_h, forward_c, backward_h, backward_c = self.bilstm(x)
        bi_output = self.fc0(bi_output)
        reduced_hidden = tf.concat([forward_h, backward_h], axis=1)
        reduced_hidden = self.fc_hidden(reduced_hidden)
        # output, hidden_state, cell_state = self.lstm(bi_output)
        # print("OUTPUT:", output.shape)
        y = self.fc1(bi_output)
        y = tf.reshape(y, (-1, x.shape[2]))
        y = self.fc2(y)
        return y, reduced_hidden, attention_weights
    
    def reset_state(self, batch_size):
        return tf.zeros((batch_size, self.units))

Training¶

In [ ]:
# Create Model
def create_model(encoder_class, decoder_class, embedding_dim, units, vocab_size):
    encoder = encoder_class(embedding_dim)
    decoder = decoder_class(embedding_dim, units, vocab_size)
    return encoder, decoder

# Tokenization and BLEU evaluation functions
def tokenize_captions(captions, tokenizer):
    return [tokenizer.texts_to_sequences([cap])[0] for cap in captions]

def decode_predictions(preds, tokenizer):
    pred_captions = []
    for pred in preds:
        decoded_sentence = []
        for idx in pred:
            if idx == tokenizer.word_index['<end>']:
                break
            decoded_sentence.append(tokenizer.index_word[idx])
        pred_captions.append(decoded_sentence)
    return pred_captions
In [ ]:
# Optimiseur ADAM
optimizer = tf.keras.optimizers.Adam() #DONE
# La fonction de perte
# SparseCategoricalCrossentropy est une fonction de perte qui est utilisée pour les problèmes de classification multi-classes. Elle est utilisée lorsque les étiquettes sont des entiers et non pas des vecteurs one-hot.
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=True, reduction='none') 

def loss_function(real, pred):
    mask = tf.math.logical_not(tf.math.equal(real, 0))
    loss_ = loss_object(real, pred)

    mask = tf.cast(mask, dtype=loss_.dtype)
    loss_ *= mask

    return tf.reduce_mean(loss_)
In [ ]:
# Validation step on separate validation dataset
loss_plot = []
@tf.function
def validation_step(img_tensor, target, encoder, decoder):
    loss = 0

    # Initialisation de l'état caché pour chaque batch
    hidden = decoder.reset_state(batch_size=target.shape[0])
    
    # Initialiser l'entrée du décodeur
    dec_input = tf.expand_dims([tokenizer.word_index['<start>']] * target.shape[0], 1)
    
    with tf.GradientTape() as tape: # Offre la possibilité de calculer le gradient du loss
        features = encoder(img_tensor)

        for i in range(1, target.shape[1]):
            # Prédiction des i'èmes mot du batch avec le décodeur
            predictions, hidden, _ = decoder(dec_input, features, hidden)
            loss += loss_function(target[:, i], predictions)

            # Le mot correct à l'étap i est donné en entrée à l'étape (i+1)
            dec_input = tf.expand_dims(target[:, i], 1)

    total_loss = (loss / int(target.shape[1])) # Calcul de la perte moyenne par mot du batch courant

    trainable_variables = encoder.trainable_variables + decoder.trainable_variables

    gradients = tape.gradient(loss, trainable_variables)

    optimizer.apply_gradients(zip(gradients, trainable_variables))

    return loss, total_loss

Model 1¶

Bidirectional LSTM with Bahdanau attention and ResNet50

In [ ]:
# Optimizer and Checkpoint Management
checkpoint_path = "./checkpoints/bi-lstm-attention"
In [ ]:
def train_model(encoder, decoder, ckpt_manager, start_epoch=0):
    # Modified training loop with BLEU score calculation
    EPOCHS = 20

    for epoch in range(start_epoch, EPOCHS):
        start = time.time()
        total_loss = 0
        
        for (batch, (img_tensor, target)) in enumerate(dataset):
            batch_loss, t_loss = validation_step(img_tensor, target, encoder, decoder)
            total_loss += t_loss

            if batch % 100 == 0:
                print ('Epoch {} Batch {} Loss {:.4f}'.format(
                epoch + 1, batch, batch_loss.numpy() / int(target.shape[1])))
        # sauvegarde de la perte
        loss_plot.append(total_loss / num_steps)

        if epoch % 5 == 0:
            ckpt_manager.save()

        print ('Epoch {} Loss {:.6f}'.format(epoch + 1,
                                            total_loss/num_steps))
        print ('Time taken for 1 epoch {} sec\n'.format(time.time() - start))

    # Affichage de la courbe d'entrainement
    plt.plot(loss_plot)
    plt.xlabel('Epochs')
    plt.ylabel('Loss')
    plt.title('Loss Plot')
    plt.show()
In [ ]:
# Function to evaluate the image captioning model
def evaluate(image, encoder, decoder):
    attention_plot = np.zeros((max_length, attention_features_shape))

    hidden = decoder.reset_state(batch_size=1)

    temp_input = tf.expand_dims(load_image(image)[0], 0)
    img_tensor_val = image_features_extract_model(temp_input)
    img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0], -1, img_tensor_val.shape[3]))

    features = encoder(img_tensor_val)

    dec_input = tf.expand_dims([tokenizer.word_index['<start>']], 0)
    result = []

    for i in range(max_length):
        predictions, hidden, attention_weights = decoder(dec_input, features, hidden)

        attention_plot[i] = tf.reshape(attention_weights, (-1, )).numpy()

        predicted_id = tf.random.categorical(predictions, 1)[0][0].numpy()
        result.append(tokenizer.index_word[predicted_id])

        if tokenizer.index_word[predicted_id] == '<end>':
            return result, attention_plot

        dec_input = tf.expand_dims([predicted_id], 0)

    return result, attention_plot

# Function to display the BLEU score
def display_bleu_score(image, result):
    references = []

    # Display image
    image_show = PIL.Image.open(image)
    plt.imshow(image_show)
    plt.axis('off')
    plt.show()

    print("\n" + "*" * 60)
    print("Predicted Caption :")
    print(' '.join(result))

    # Find the corresponding references (captions) for this specific image
        # Find the corresponding references (captions) for this specific image
    for i, img_name in enumerate(img_name_val):
        if img_name == image:  # Compare with the image being processed
            ref = []
            for token in cap_val[i]:
                if token != 0:  # Ignore padding (0)
                    word = tokenizer.index_word.get(token, '<unk>')  # Handle missing tokens
                    ref.append(word)
            references.append(ref)
    print("\n" + "*" * 60)
    print("References :")
    for ref in references[:5]:
        print(' '.join(ref))

    print("...")

    # Convert references for BLEU score evaluation
    references_tokenized = [references]  # Multiple references for a single image

    # Since we're calculating BLEU for a single image, ensure predictions are used as a single list
    predictions = [result]  # Make the predicted caption a single hypothesis list

    # Calculate BLEU scores
    bleu_1 = corpus_bleu(references_tokenized, predictions, weights=(1.0, 0, 0, 0))
    bleu_2 = corpus_bleu(references_tokenized, predictions, weights=(0.5, 0.5, 0, 0))
    bleu_3 = corpus_bleu(references_tokenized, predictions, weights=(0.33, 0.33, 0.33, 0))
    bleu_4 = corpus_bleu(references_tokenized, predictions, weights=(0.25, 0.25, 0.25, 0.25))

    # Print BLEU scores
    print("\n" + "*" * 60)
    print(f"BLEU Score :")
    print(f"unigram  = {bleu_1:.10f}")
    print(f"bigram   = {bleu_2:.10f}")
    print(f"trigram  = {bleu_3:.10f}")
    print(f"4-gram = {bleu_4:.10f}")
    print("*" * 60)

# Function for visualizing attention on the image
def plot_attention(image, result, attention_plot):
    temp_image = np.array(PIL.Image.open(image))

    fig = plt.figure(figsize=(10, 10))

    len_result = len(result)
    for l in range(len_result):
        temp_att = np.resize(attention_plot[l], (8, 8))
        ax = fig.add_subplot(len_result // 2, len_result // 2, l + 1)
        ax.set_title(result[l])
        img = ax.imshow(temp_image)
        ax.imshow(temp_att, cmap='gray', alpha=0.6, extent=img.get_extent())

    plt.tight_layout()
    plt.show()
In [ ]:
# Instantiate encoder and decoder
encoder, decoder = create_model(CNN_Encoder, RNN_Decoder_Bidirectional_LSTM, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder, 
                           decoder=decoder, 
                           optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt, 
                                          checkpoint_path, 
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
    if ckpt_manager.latest_checkpoint:
        print(f"Restored from {ckpt_manager.latest_checkpoint}")
        print(f"Resuming training from epoch {start_epoch}")
Restored from ./checkpoints/bi-lstm-attention/ckpt-7
Resuming training from epoch 7
In [ ]:
train_model(encoder, decoder, ckpt_manager)
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.0
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.16
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.17
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.18
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.21
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).encoder.fc._kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.0
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.16
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.17
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.18
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.19
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.20
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.21
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._trainable_variables.22
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/keras/src/optimizers/base_optimizer.py:731: UserWarning: Gradients do not exist for variables ['kernel', 'kernel', 'kernel', 'bias', 'kernel'] when minimizing the loss. If using `model.compile()`, did you forget to provide a `loss` argument?
  warnings.warn(
Epoch 1 Batch 0 Loss 1.3308
Epoch 1 Batch 100 Loss 0.6665
Epoch 1 Batch 200 Loss 0.6181
Epoch 1 Batch 300 Loss 0.6507
Epoch 1 Batch 400 Loss 0.6929
Epoch 1 Batch 500 Loss 0.5760
Epoch 1 Batch 600 Loss 0.6114
Epoch 1 Batch 700 Loss 0.6629
Epoch 1 Batch 800 Loss 0.8464
Epoch 1 Batch 900 Loss 0.7834
Epoch 1 Batch 1000 Loss 0.7182
Epoch 1 Batch 1100 Loss 0.5745
Epoch 1 Batch 1200 Loss 0.5548
2024-10-23 14:26:13.988737: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 0.661850
Time taken for 1 epoch 451.702716588974 sec

Epoch 2 Batch 0 Loss 0.6918
Epoch 2 Batch 100 Loss 0.6655
Epoch 2 Batch 200 Loss 0.5469
Epoch 2 Batch 300 Loss 0.6484
Epoch 2 Batch 400 Loss 0.7081
Epoch 2 Batch 500 Loss 0.5811
Epoch 2 Batch 600 Loss 0.6283
Epoch 2 Batch 700 Loss 0.5951
Epoch 2 Batch 800 Loss 0.5884
Epoch 2 Batch 900 Loss 0.6524
Epoch 2 Batch 1000 Loss 0.5944
Epoch 2 Batch 1100 Loss 0.6432
Epoch 2 Batch 1200 Loss 0.5875
Epoch 2 Loss 0.588800
Time taken for 1 epoch 404.31619668006897 sec

Epoch 3 Batch 0 Loss 0.6631
Epoch 3 Batch 100 Loss 0.5700
Epoch 3 Batch 200 Loss 0.6315
Epoch 3 Batch 300 Loss 0.5168
Epoch 3 Batch 400 Loss 0.6637
Epoch 3 Batch 500 Loss 0.5791
Epoch 3 Batch 600 Loss 0.5642
Epoch 3 Batch 700 Loss 0.5799
Epoch 3 Batch 800 Loss 0.5303
Epoch 3 Batch 900 Loss 0.5454
Epoch 3 Batch 1000 Loss 0.5725
Epoch 3 Batch 1100 Loss 0.5547
Epoch 3 Batch 1200 Loss 0.5921
2024-10-23 14:39:42.814140: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.548848
Time taken for 1 epoch 404.26971983909607 sec

Epoch 4 Batch 0 Loss 0.5791
Epoch 4 Batch 100 Loss 0.5735
Epoch 4 Batch 200 Loss 0.5292
Epoch 4 Batch 300 Loss 0.6178
Epoch 4 Batch 400 Loss 0.6346
Epoch 4 Batch 500 Loss 0.4662
Epoch 4 Batch 600 Loss 0.5170
Epoch 4 Batch 700 Loss 0.5487
Epoch 4 Batch 800 Loss 0.5870
Epoch 4 Batch 900 Loss 0.5018
Epoch 4 Batch 1000 Loss 0.4845
Epoch 4 Batch 1100 Loss 0.4999
Epoch 4 Batch 1200 Loss 0.5039
Epoch 4 Loss 0.514951
Time taken for 1 epoch 404.21688580513 sec

Epoch 5 Batch 0 Loss 0.5935
Epoch 5 Batch 100 Loss 0.4677
Epoch 5 Batch 200 Loss 0.5188
Epoch 5 Batch 300 Loss 0.4835
Epoch 5 Batch 400 Loss 0.5191
Epoch 5 Batch 500 Loss 0.5512
Epoch 5 Batch 600 Loss 0.5217
Epoch 5 Batch 700 Loss 0.4792
Epoch 5 Batch 800 Loss 0.4835
Epoch 5 Batch 900 Loss 0.5592
Epoch 5 Batch 1000 Loss 0.4922
Epoch 5 Batch 1100 Loss 0.4928
Epoch 5 Batch 1200 Loss 0.5193
Epoch 5 Loss 0.483366
Time taken for 1 epoch 404.2356188297272 sec

Epoch 6 Batch 0 Loss 0.5725
Epoch 6 Batch 100 Loss 0.4082
Epoch 6 Batch 200 Loss 0.4265
Epoch 6 Batch 300 Loss 0.4496
Epoch 6 Batch 400 Loss 0.4361
Epoch 6 Batch 500 Loss 0.5292
Epoch 6 Batch 600 Loss 0.3833
Epoch 6 Batch 700 Loss 0.5077
Epoch 6 Batch 800 Loss 0.4557
Epoch 6 Batch 900 Loss 0.5049
Epoch 6 Batch 1000 Loss 0.3994
Epoch 6 Batch 1100 Loss 0.4148
Epoch 6 Batch 1200 Loss 0.4237
Epoch 6 Loss 0.455444
Time taken for 1 epoch 404.40891885757446 sec

Epoch 7 Batch 0 Loss 0.4574
Epoch 7 Batch 100 Loss 0.5356
Epoch 7 Batch 200 Loss 0.4975
Epoch 7 Batch 300 Loss 0.3896
Epoch 7 Batch 400 Loss 0.4934
Epoch 7 Batch 500 Loss 0.4106
Epoch 7 Batch 600 Loss 0.4141
Epoch 7 Batch 700 Loss 0.4749
Epoch 7 Batch 800 Loss 0.4093
Epoch 7 Batch 900 Loss 0.5058
Epoch 7 Batch 1000 Loss 0.4484
Epoch 7 Batch 1100 Loss 0.3890
Epoch 7 Batch 1200 Loss 0.3996
2024-10-23 15:06:39.630645: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.431582
Time taken for 1 epoch 403.9550528526306 sec

Epoch 8 Batch 0 Loss 0.4493
Epoch 8 Batch 100 Loss 0.4381
Epoch 8 Batch 200 Loss 0.4439
Epoch 8 Batch 300 Loss 0.4573
Epoch 8 Batch 400 Loss 0.4233
Epoch 8 Batch 500 Loss 0.4287
Epoch 8 Batch 600 Loss 0.3951
Epoch 8 Batch 700 Loss 0.4274
Epoch 8 Batch 800 Loss 0.3688
Epoch 8 Batch 900 Loss 0.3192
Epoch 8 Batch 1000 Loss 0.3349
Epoch 8 Batch 1100 Loss 0.4521
Epoch 8 Batch 1200 Loss 0.3516
Epoch 8 Loss 0.409525
Time taken for 1 epoch 404.1138606071472 sec

Epoch 9 Batch 0 Loss 0.3865
Epoch 9 Batch 100 Loss 0.3840
Epoch 9 Batch 200 Loss 0.4069
Epoch 9 Batch 300 Loss 0.3551
Epoch 9 Batch 400 Loss 0.4455
Epoch 9 Batch 500 Loss 0.3898
Epoch 9 Batch 600 Loss 0.3437
Epoch 9 Batch 700 Loss 0.3843
Epoch 9 Batch 800 Loss 0.3510
Epoch 9 Batch 900 Loss 0.3901
Epoch 9 Batch 1000 Loss 0.3577
Epoch 9 Batch 1100 Loss 0.3831
Epoch 9 Batch 1200 Loss 0.3395
Epoch 9 Loss 0.389218
Time taken for 1 epoch 403.94647097587585 sec

Epoch 10 Batch 0 Loss 0.4170
Epoch 10 Batch 100 Loss 0.3671
Epoch 10 Batch 200 Loss 0.3773
Epoch 10 Batch 300 Loss 0.3489
Epoch 10 Batch 400 Loss 0.3824
Epoch 10 Batch 500 Loss 0.2920
Epoch 10 Batch 600 Loss 0.3427
Epoch 10 Batch 700 Loss 0.4106
Epoch 10 Batch 800 Loss 0.3972
Epoch 10 Batch 900 Loss 0.3768
Epoch 10 Batch 1000 Loss 0.3519
Epoch 10 Batch 1100 Loss 0.3765
Epoch 10 Batch 1200 Loss 0.4347
Epoch 10 Loss 0.371290
Time taken for 1 epoch 404.24859046936035 sec

Epoch 11 Batch 0 Loss 0.3070
Epoch 11 Batch 100 Loss 0.3257
Epoch 11 Batch 200 Loss 0.3190
Epoch 11 Batch 300 Loss 0.3252
Epoch 11 Batch 400 Loss 0.3601
Epoch 11 Batch 500 Loss 0.3623
Epoch 11 Batch 600 Loss 0.3718
Epoch 11 Batch 700 Loss 0.3099
Epoch 11 Batch 800 Loss 0.3237
Epoch 11 Batch 900 Loss 0.4156
Epoch 11 Batch 1000 Loss 0.3426
Epoch 11 Batch 1100 Loss 0.3340
Epoch 11 Batch 1200 Loss 0.3582
Epoch 11 Loss 0.357097
Time taken for 1 epoch 404.46499466896057 sec

Epoch 12 Batch 0 Loss 0.3334
Epoch 12 Batch 100 Loss 0.4213
Epoch 12 Batch 200 Loss 0.3404
Epoch 12 Batch 300 Loss 0.3463
Epoch 12 Batch 400 Loss 0.2856
Epoch 12 Batch 500 Loss 0.3015
Epoch 12 Batch 600 Loss 0.3505
Epoch 12 Batch 700 Loss 0.3681
Epoch 12 Batch 800 Loss 0.3821
Epoch 12 Batch 900 Loss 0.3475
Epoch 12 Batch 1000 Loss 0.3103
Epoch 12 Batch 1100 Loss 0.3185
Epoch 12 Batch 1200 Loss 0.3402
Epoch 12 Loss 0.344337
Time taken for 1 epoch 403.9861605167389 sec

Epoch 13 Batch 0 Loss 0.3339
Epoch 13 Batch 100 Loss 0.3700
Epoch 13 Batch 200 Loss 0.3345
Epoch 13 Batch 300 Loss 0.3919
Epoch 13 Batch 400 Loss 0.3828
Epoch 13 Batch 500 Loss 0.3937
Epoch 13 Batch 600 Loss 0.3208
Epoch 13 Batch 700 Loss 0.3639
Epoch 13 Batch 800 Loss 0.3492
Epoch 13 Batch 900 Loss 0.3345
Epoch 13 Batch 1000 Loss 0.2932
Epoch 13 Batch 1100 Loss 0.3440
Epoch 13 Batch 1200 Loss 0.2918
Epoch 13 Loss 0.329917
Time taken for 1 epoch 403.9970133304596 sec

Epoch 14 Batch 0 Loss 0.2767
Epoch 14 Batch 100 Loss 0.3617
Epoch 14 Batch 200 Loss 0.4307
Epoch 14 Batch 300 Loss 0.3563
Epoch 14 Batch 400 Loss 0.3126
Epoch 14 Batch 500 Loss 0.3392
Epoch 14 Batch 600 Loss 0.2766
Epoch 14 Batch 700 Loss 0.3071
Epoch 14 Batch 800 Loss 0.3272
Epoch 14 Batch 900 Loss 0.3577
Epoch 14 Batch 1000 Loss 0.2934
Epoch 14 Batch 1100 Loss 0.3177
Epoch 14 Batch 1200 Loss 0.2823
Epoch 14 Loss 0.320054
Time taken for 1 epoch 403.80691361427307 sec

Epoch 15 Batch 0 Loss 0.2944
Epoch 15 Batch 100 Loss 0.3272
Epoch 15 Batch 200 Loss 0.2953
Epoch 15 Batch 300 Loss 0.2562
Epoch 15 Batch 400 Loss 0.3174
Epoch 15 Batch 500 Loss 0.2696
Epoch 15 Batch 600 Loss 0.2852
Epoch 15 Batch 700 Loss 0.3181
Epoch 15 Batch 800 Loss 0.3468
Epoch 15 Batch 900 Loss 0.3092
Epoch 15 Batch 1000 Loss 0.3094
Epoch 15 Batch 1100 Loss 0.3228
Epoch 15 Batch 1200 Loss 0.3343
2024-10-23 16:00:32.177810: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.308469
Time taken for 1 epoch 403.98316717147827 sec

Epoch 16 Batch 0 Loss 0.2629
Epoch 16 Batch 100 Loss 0.3160
Epoch 16 Batch 200 Loss 0.2330
Epoch 16 Batch 300 Loss 0.2633
Epoch 16 Batch 400 Loss 0.2905
Epoch 16 Batch 500 Loss 0.2991
Epoch 16 Batch 600 Loss 0.3089
Epoch 16 Batch 700 Loss 0.2852
Epoch 16 Batch 800 Loss 0.3773
Epoch 16 Batch 900 Loss 0.2933
Epoch 16 Batch 1000 Loss 0.2383
Epoch 16 Batch 1100 Loss 0.3361
Epoch 16 Batch 1200 Loss 0.2893
Epoch 16 Loss 0.299219
Time taken for 1 epoch 404.25311255455017 sec

Epoch 17 Batch 0 Loss 0.2630
Epoch 17 Batch 100 Loss 0.3076
Epoch 17 Batch 200 Loss 0.2883
Epoch 17 Batch 300 Loss 0.2994
Epoch 17 Batch 400 Loss 0.2915
Epoch 17 Batch 500 Loss 0.2792
Epoch 17 Batch 600 Loss 0.2767
Epoch 17 Batch 700 Loss 0.3116
Epoch 17 Batch 800 Loss 0.2486
Epoch 17 Batch 900 Loss 0.3119
Epoch 17 Batch 1000 Loss 0.3007
Epoch 17 Batch 1100 Loss 0.3087
Epoch 17 Batch 1200 Loss 0.2493
Epoch 17 Loss 0.290037
Time taken for 1 epoch 404.0419964790344 sec

Epoch 18 Batch 0 Loss 0.2740
Epoch 18 Batch 100 Loss 0.3151
Epoch 18 Batch 200 Loss 0.2336
Epoch 18 Batch 300 Loss 0.3689
Epoch 18 Batch 400 Loss 0.2776
Epoch 18 Batch 500 Loss 0.2854
Epoch 18 Batch 600 Loss 0.3365
Epoch 18 Batch 700 Loss 0.2895
Epoch 18 Batch 800 Loss 0.3202
Epoch 18 Batch 900 Loss 0.2926
Epoch 18 Batch 1000 Loss 0.2297
Epoch 18 Batch 1100 Loss 0.2938
Epoch 18 Batch 1200 Loss 0.2719
Epoch 18 Loss 0.283209
Time taken for 1 epoch 403.9328944683075 sec

Epoch 19 Batch 0 Loss 0.2679
Epoch 19 Batch 100 Loss 0.2579
Epoch 19 Batch 200 Loss 0.2724
Epoch 19 Batch 300 Loss 0.2480
Epoch 19 Batch 400 Loss 0.2536
Epoch 19 Batch 500 Loss 0.2941
Epoch 19 Batch 600 Loss 0.2518
Epoch 19 Batch 700 Loss 0.3054
Epoch 19 Batch 800 Loss 0.2742
Epoch 19 Batch 900 Loss 0.2620
Epoch 19 Batch 1000 Loss 0.2580
Epoch 19 Batch 1100 Loss 0.2310
Epoch 19 Batch 1200 Loss 0.2654
Epoch 19 Loss 0.275977
Time taken for 1 epoch 403.85397386550903 sec

Epoch 20 Batch 0 Loss 0.2785
Epoch 20 Batch 100 Loss 0.2603
Epoch 20 Batch 200 Loss 0.2809
Epoch 20 Batch 300 Loss 0.2486
Epoch 20 Batch 400 Loss 0.2204
Epoch 20 Batch 500 Loss 0.2866
Epoch 20 Batch 600 Loss 0.3094
Epoch 20 Batch 700 Loss 0.2318
Epoch 20 Batch 800 Loss 0.2738
Epoch 20 Batch 900 Loss 0.2883
Epoch 20 Batch 1000 Loss 0.2491
Epoch 20 Batch 1100 Loss 0.2722
Epoch 20 Batch 1200 Loss 0.2617
Epoch 20 Loss 0.269723
Time taken for 1 epoch 403.97234559059143 sec

No description has been provided for this image
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder, decoder)
    
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
./train2014/COCO_train2014_000000314787.jpg
Real Caption: <start> a bathroom with brick walls a toilet and sink <end>
Prediction Caption: a very stylish bathroom near an <unk> pan during everything even <unk> pan position <end>
No description has been provided for this image
In [ ]:
display_bleu_score(image, result)
No description has been provided for this image
************************************************************
Predicted Caption :
a newly painted sterile bathroom with the door <end>

************************************************************
References :
<start> a white bath tub sitting next to a toilet <end>
<start> a tub and toilet in a small bathroom <end>
<start> a bathroom with some beige walls and a brown cabinet <end>
<start> a small bathroom that has a sink and a toilet <end>
<start> entrance to bathroom with shower bathtub sink with vanity and white toilet commode <end>
...

************************************************************
BLEU Score :
unigram  = 0.3977063630
bigram   = 0.2109156497
trigram  = 0.0000000000
4-gram = 0.0000000000
************************************************************
In [ ]:
# play frequency sound when training is done (linux sound)
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Save the Encoder model
encoder.save('models/captioning_models/encoder_resnet_bilstm_model.keras')

# Save the Decoder model
decoder.save('models/captioning_models/decoder1_resnet_bilstm_model.keras')
In [ ]:
def evaluate_average_bleu(encoder, decoder, img_name_val, cap_val, tokenizer):
    total_bleu_1 = 0
    total_bleu_2 = 0
    total_bleu_3 = 0
    total_bleu_4 = 0
    num_samples = len(img_name_val)
    
    for i in range(num_samples):
        image = img_name_val[i]
        
        # Get real caption
        real_caption = [[tokenizer.index_word.get(idx, '<unk>') for idx in cap_val[i] if idx not in [0]]]
        
        # Get the predicted caption
        result, _ = evaluate(image, encoder, decoder)
        
        # Convert result into format for BLEU score
        result_caption = [result]
        
        # Calculate BLEU scores
        bleu_1 = corpus_bleu(real_caption, result_caption, weights=(1.0, 0, 0, 0))
        bleu_2 = corpus_bleu(real_caption, result_caption, weights=(0.5, 0.5, 0, 0))
        bleu_3 = corpus_bleu(real_caption, result_caption, weights=(0.33, 0.33, 0.33, 0))
        bleu_4 = corpus_bleu(real_caption, result_caption, weights=(0.25, 0.25, 0.25, 0.25))
        
        # Accumulate BLEU scores
        total_bleu_1 += bleu_1
        total_bleu_2 += bleu_2
        total_bleu_3 += bleu_3
        total_bleu_4 += bleu_4
        
        # Optional: print progress every 500 images
        if i % 500 == 0:
            print(f"Processed {i}/{num_samples} images")

    # Calculate the average BLEU score for each n-gram
    avg_bleu_1 = total_bleu_1 / num_samples
    avg_bleu_2 = total_bleu_2 / num_samples
    avg_bleu_3 = total_bleu_3 / num_samples
    avg_bleu_4 = total_bleu_4 / num_samples

    # Print the average BLEU scores
    print("\n" + "*" * 60)
    print(f"Average BLEU Score on Validation Dataset:")
    print(f"unigram  = {avg_bleu_1:.10f}")
    print(f"bigram   = {avg_bleu_2:.10f}")
    print(f"trigram  = {avg_bleu_3:.10f}")
    print(f"4-gram = {avg_bleu_4:.10f}")
    print("*" * 60)
    
    return avg_bleu_1, avg_bleu_2, avg_bleu_3, avg_bleu_4
In [ ]:
# Evaluate the model on the validation dataset
evaluate_average_bleu(encoder, decoder, img_name_val, cap_val, tokenizer)
#
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.0994896786
bigram   = 0.0000000000
trigram  = 0.0000000000
4-gram = 0.0000000000
************************************************************
(0.09948967855497265,
 4.364591788851547e-155,
 3.806644534300383e-204,
 9.343453754705764e-232)

Model 2¶

Single Layer GRU with BLEU and ResNet50

In [ ]:
# Instantiate encoder and decoder
encoder_gru_resnet, decoder_gru_resnet = create_model(CNN_Encoder, RNN_Decoder_GRU, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_resnet, 
                           decoder=decoder_gru_resnet, 
                           optimizer=optimizer)
checkpoint_path = "./checkpoints/gru-attention"
ckpt_manager = tf.train.CheckpointManager(ckpt, 
                                          checkpoint_path, 
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_resnet, decoder_gru_resnet, ckpt_manager)
Epoch 1 Batch 0 Loss 2.0964
Epoch 1 Batch 100 Loss 1.1680
Epoch 1 Batch 200 Loss 0.9441
Epoch 1 Batch 300 Loss 1.0024
Epoch 1 Batch 400 Loss 1.0602
Epoch 1 Batch 500 Loss 1.0049
Epoch 1 Batch 600 Loss 0.8326
Epoch 1 Batch 700 Loss 0.7792
Epoch 1 Batch 800 Loss 0.8974
Epoch 1 Batch 900 Loss 0.7943
Epoch 1 Batch 1000 Loss 0.9234
Epoch 1 Batch 1100 Loss 0.8752
Epoch 1 Batch 1200 Loss 0.8474
2024-10-23 19:13:16.528266: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 0.943236
Time taken for 1 epoch 234.17758750915527 sec

Epoch 2 Batch 0 Loss 1.0063
Epoch 2 Batch 100 Loss 0.9182
Epoch 2 Batch 200 Loss 0.8619
Epoch 2 Batch 300 Loss 0.7751
Epoch 2 Batch 400 Loss 0.8642
Epoch 2 Batch 500 Loss 0.8193
Epoch 2 Batch 600 Loss 0.7891
Epoch 2 Batch 700 Loss 0.7564
Epoch 2 Batch 800 Loss 0.7790
Epoch 2 Batch 900 Loss 0.6983
Epoch 2 Batch 1000 Loss 0.8320
Epoch 2 Batch 1100 Loss 0.7041
Epoch 2 Batch 1200 Loss 0.6296
Epoch 2 Loss 0.790500
Time taken for 1 epoch 176.20997762680054 sec

Epoch 3 Batch 0 Loss 0.7597
Epoch 3 Batch 100 Loss 0.7468
Epoch 3 Batch 200 Loss 0.6478
Epoch 3 Batch 300 Loss 0.7488
Epoch 3 Batch 400 Loss 0.7710
Epoch 3 Batch 500 Loss 0.6730
Epoch 3 Batch 600 Loss 0.6807
Epoch 3 Batch 700 Loss 0.6980
Epoch 3 Batch 800 Loss 0.7633
Epoch 3 Batch 900 Loss 0.6081
Epoch 3 Batch 1000 Loss 0.5709
Epoch 3 Batch 1100 Loss 0.7082
Epoch 3 Batch 1200 Loss 0.7064
2024-10-23 19:19:09.036911: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.728305
Time taken for 1 epoch 176.13086009025574 sec

Epoch 4 Batch 0 Loss 0.7767
Epoch 4 Batch 100 Loss 0.7977
Epoch 4 Batch 200 Loss 0.7996
Epoch 4 Batch 300 Loss 0.6901
Epoch 4 Batch 400 Loss 0.9279
Epoch 4 Batch 500 Loss 0.6875
Epoch 4 Batch 600 Loss 0.6675
Epoch 4 Batch 700 Loss 0.6429
Epoch 4 Batch 800 Loss 0.5904
Epoch 4 Batch 900 Loss 0.7088
Epoch 4 Batch 1000 Loss 0.6508
Epoch 4 Batch 1100 Loss 0.5496
Epoch 4 Batch 1200 Loss 0.5898
Epoch 4 Loss 0.681548
Time taken for 1 epoch 176.12795042991638 sec

Epoch 5 Batch 0 Loss 0.7248
Epoch 5 Batch 100 Loss 0.5946
Epoch 5 Batch 200 Loss 0.6493
Epoch 5 Batch 300 Loss 0.6985
Epoch 5 Batch 400 Loss 0.6287
Epoch 5 Batch 500 Loss 0.6723
Epoch 5 Batch 600 Loss 0.6206
Epoch 5 Batch 700 Loss 0.6803
Epoch 5 Batch 800 Loss 0.6481
Epoch 5 Batch 900 Loss 0.5847
Epoch 5 Batch 1000 Loss 0.6799
Epoch 5 Batch 1100 Loss 0.7091
Epoch 5 Batch 1200 Loss 0.6551
Epoch 5 Loss 0.642496
Time taken for 1 epoch 175.99731302261353 sec

Epoch 6 Batch 0 Loss 0.6666
Epoch 6 Batch 100 Loss 0.6303
Epoch 6 Batch 200 Loss 0.6445
Epoch 6 Batch 300 Loss 0.6242
Epoch 6 Batch 400 Loss 0.4923
Epoch 6 Batch 500 Loss 0.4911
Epoch 6 Batch 600 Loss 0.6148
Epoch 6 Batch 700 Loss 0.6318
Epoch 6 Batch 800 Loss 0.5802
Epoch 6 Batch 900 Loss 0.6092
Epoch 6 Batch 1000 Loss 0.6573
Epoch 6 Batch 1100 Loss 0.6447
Epoch 6 Batch 1200 Loss 0.5740
Epoch 6 Loss 0.606568
Time taken for 1 epoch 176.03090715408325 sec

Epoch 7 Batch 0 Loss 0.6503
Epoch 7 Batch 100 Loss 0.5569
Epoch 7 Batch 200 Loss 0.6204
Epoch 7 Batch 300 Loss 0.5214
Epoch 7 Batch 400 Loss 0.6260
Epoch 7 Batch 500 Loss 0.6618
Epoch 7 Batch 600 Loss 0.6330
Epoch 7 Batch 700 Loss 0.5900
Epoch 7 Batch 800 Loss 0.5842
Epoch 7 Batch 900 Loss 0.5461
Epoch 7 Batch 1000 Loss 0.5304
Epoch 7 Batch 1100 Loss 0.5495
Epoch 7 Batch 1200 Loss 0.5432
2024-10-23 19:30:53.239752: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.573152
Time taken for 1 epoch 176.04660749435425 sec

Epoch 8 Batch 0 Loss 0.6648
Epoch 8 Batch 100 Loss 0.5965
Epoch 8 Batch 200 Loss 0.5658
Epoch 8 Batch 300 Loss 0.5529
Epoch 8 Batch 400 Loss 0.6312
Epoch 8 Batch 500 Loss 0.5119
Epoch 8 Batch 600 Loss 0.4763
Epoch 8 Batch 700 Loss 0.5234
Epoch 8 Batch 800 Loss 0.5312
Epoch 8 Batch 900 Loss 0.5735
Epoch 8 Batch 1000 Loss 0.4996
Epoch 8 Batch 1100 Loss 0.5005
Epoch 8 Batch 1200 Loss 0.5560
Epoch 8 Loss 0.541674
Time taken for 1 epoch 176.0696837902069 sec

Epoch 9 Batch 0 Loss 0.5775
Epoch 9 Batch 100 Loss 0.4871
Epoch 9 Batch 200 Loss 0.5153
Epoch 9 Batch 300 Loss 0.4783
Epoch 9 Batch 400 Loss 0.4683
Epoch 9 Batch 500 Loss 0.5768
Epoch 9 Batch 600 Loss 0.4385
Epoch 9 Batch 700 Loss 0.4511
Epoch 9 Batch 800 Loss 0.5406
Epoch 9 Batch 900 Loss 0.5116
Epoch 9 Batch 1000 Loss 0.5508
Epoch 9 Batch 1100 Loss 0.4673
Epoch 9 Batch 1200 Loss 0.4499
Epoch 9 Loss 0.512716
Time taken for 1 epoch 175.94577312469482 sec

Epoch 10 Batch 0 Loss 0.4741
Epoch 10 Batch 100 Loss 0.4460
Epoch 10 Batch 200 Loss 0.5673
Epoch 10 Batch 300 Loss 0.5252
Epoch 10 Batch 400 Loss 0.4914
Epoch 10 Batch 500 Loss 0.4032
Epoch 10 Batch 600 Loss 0.5410
Epoch 10 Batch 700 Loss 0.5118
Epoch 10 Batch 800 Loss 0.4926
Epoch 10 Batch 900 Loss 0.4248
Epoch 10 Batch 1000 Loss 0.4502
Epoch 10 Batch 1100 Loss 0.5084
Epoch 10 Batch 1200 Loss 0.3949
Epoch 10 Loss 0.484986
Time taken for 1 epoch 175.99557375907898 sec

Epoch 11 Batch 0 Loss 0.3847
Epoch 11 Batch 100 Loss 0.4137
Epoch 11 Batch 200 Loss 0.3729
Epoch 11 Batch 300 Loss 0.4496
Epoch 11 Batch 400 Loss 0.4737
Epoch 11 Batch 500 Loss 0.4954
Epoch 11 Batch 600 Loss 0.5368
Epoch 11 Batch 700 Loss 0.5587
Epoch 11 Batch 800 Loss 0.5428
Epoch 11 Batch 900 Loss 0.4295
Epoch 11 Batch 1000 Loss 0.4540
Epoch 11 Batch 1100 Loss 0.4445
Epoch 11 Batch 1200 Loss 0.5133
Epoch 11 Loss 0.460534
Time taken for 1 epoch 176.05313158035278 sec

Epoch 12 Batch 0 Loss 0.5090
Epoch 12 Batch 100 Loss 0.3912
Epoch 12 Batch 200 Loss 0.4012
Epoch 12 Batch 300 Loss 0.4587
Epoch 12 Batch 400 Loss 0.4540
Epoch 12 Batch 500 Loss 0.3738
Epoch 12 Batch 600 Loss 0.4764
Epoch 12 Batch 700 Loss 0.5041
Epoch 12 Batch 800 Loss 0.4003
Epoch 12 Batch 900 Loss 0.4102
Epoch 12 Batch 1000 Loss 0.4880
Epoch 12 Batch 1100 Loss 0.4182
Epoch 12 Batch 1200 Loss 0.4300
Epoch 12 Loss 0.437338
Time taken for 1 epoch 176.01339483261108 sec

Epoch 13 Batch 0 Loss 0.4201
Epoch 13 Batch 100 Loss 0.3929
Epoch 13 Batch 200 Loss 0.4307
Epoch 13 Batch 300 Loss 0.3881
Epoch 13 Batch 400 Loss 0.4916
Epoch 13 Batch 500 Loss 0.4010
Epoch 13 Batch 600 Loss 0.3226
Epoch 13 Batch 700 Loss 0.3739
Epoch 13 Batch 800 Loss 0.4380
Epoch 13 Batch 900 Loss 0.3780
Epoch 13 Batch 1000 Loss 0.4192
Epoch 13 Batch 1100 Loss 0.4132
Epoch 13 Batch 1200 Loss 0.3852
Epoch 13 Loss 0.416540
Time taken for 1 epoch 175.97470211982727 sec

Epoch 14 Batch 0 Loss 0.3983
Epoch 14 Batch 100 Loss 0.3778
Epoch 14 Batch 200 Loss 0.4336
Epoch 14 Batch 300 Loss 0.5072
Epoch 14 Batch 400 Loss 0.4587
Epoch 14 Batch 500 Loss 0.4077
Epoch 14 Batch 600 Loss 0.3997
Epoch 14 Batch 700 Loss 0.4283
Epoch 14 Batch 800 Loss 0.3579
Epoch 14 Batch 900 Loss 0.4148
Epoch 14 Batch 1000 Loss 0.4393
Epoch 14 Batch 1100 Loss 0.3846
Epoch 14 Batch 1200 Loss 0.4251
Epoch 14 Loss 0.397899
Time taken for 1 epoch 175.8241903781891 sec

Epoch 15 Batch 0 Loss 0.3710
Epoch 15 Batch 100 Loss 0.3208
Epoch 15 Batch 200 Loss 0.3491
Epoch 15 Batch 300 Loss 0.3646
Epoch 15 Batch 400 Loss 0.3877
Epoch 15 Batch 500 Loss 0.3793
Epoch 15 Batch 600 Loss 0.3434
Epoch 15 Batch 700 Loss 0.4042
Epoch 15 Batch 800 Loss 0.4302
Epoch 15 Batch 900 Loss 0.4447
Epoch 15 Batch 1000 Loss 0.3603
Epoch 15 Batch 1100 Loss 0.3325
Epoch 15 Batch 1200 Loss 0.4446
2024-10-23 19:54:20.998139: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.380666
Time taken for 1 epoch 175.88196086883545 sec

Epoch 16 Batch 0 Loss 0.3584
Epoch 16 Batch 100 Loss 0.3380
Epoch 16 Batch 200 Loss 0.3790
Epoch 16 Batch 300 Loss 0.4311
Epoch 16 Batch 400 Loss 0.3487
Epoch 16 Batch 500 Loss 0.3701
Epoch 16 Batch 600 Loss 0.4060
Epoch 16 Batch 700 Loss 0.3413
Epoch 16 Batch 800 Loss 0.4320
Epoch 16 Batch 900 Loss 0.3482
Epoch 16 Batch 1000 Loss 0.3454
Epoch 16 Batch 1100 Loss 0.3557
Epoch 16 Batch 1200 Loss 0.4170
Epoch 16 Loss 0.365070
Time taken for 1 epoch 176.03572297096252 sec

Epoch 17 Batch 0 Loss 0.3553
Epoch 17 Batch 100 Loss 0.3576
Epoch 17 Batch 200 Loss 0.3852
Epoch 17 Batch 300 Loss 0.3946
Epoch 17 Batch 400 Loss 0.3786
Epoch 17 Batch 500 Loss 0.3256
Epoch 17 Batch 600 Loss 0.3781
Epoch 17 Batch 700 Loss 0.3641
Epoch 17 Batch 800 Loss 0.3115
Epoch 17 Batch 900 Loss 0.3560
Epoch 17 Batch 1000 Loss 0.3803
Epoch 17 Batch 1100 Loss 0.3643
Epoch 17 Batch 1200 Loss 0.3805
Epoch 17 Loss 0.351258
Time taken for 1 epoch 175.90720772743225 sec

Epoch 18 Batch 0 Loss 0.3787
Epoch 18 Batch 100 Loss 0.3134
Epoch 18 Batch 200 Loss 0.3316
Epoch 18 Batch 300 Loss 0.3373
Epoch 18 Batch 400 Loss 0.2838
Epoch 18 Batch 500 Loss 0.3490
Epoch 18 Batch 600 Loss 0.3052
Epoch 18 Batch 700 Loss 0.2989
Epoch 18 Batch 800 Loss 0.3126
Epoch 18 Batch 900 Loss 0.3357
Epoch 18 Batch 1000 Loss 0.3329
Epoch 18 Batch 1100 Loss 0.3369
Epoch 18 Batch 1200 Loss 0.3137
Epoch 18 Loss 0.338591
Time taken for 1 epoch 176.1427402496338 sec

Epoch 19 Batch 0 Loss 0.3480
Epoch 19 Batch 100 Loss 0.2945
Epoch 19 Batch 200 Loss 0.3361
Epoch 19 Batch 300 Loss 0.3796
Epoch 19 Batch 400 Loss 0.3780
Epoch 19 Batch 500 Loss 0.2687
Epoch 19 Batch 600 Loss 0.3110
Epoch 19 Batch 700 Loss 0.3021
Epoch 19 Batch 800 Loss 0.3345
Epoch 19 Batch 900 Loss 0.3549
Epoch 19 Batch 1000 Loss 0.3258
Epoch 19 Batch 1100 Loss 0.2691
Epoch 19 Batch 1200 Loss 0.4158
Epoch 19 Loss 0.327115
Time taken for 1 epoch 176.8774073123932 sec

Epoch 20 Batch 0 Loss 0.3616
Epoch 20 Batch 100 Loss 0.3117
Epoch 20 Batch 200 Loss 0.3318
Epoch 20 Batch 300 Loss 0.2756
Epoch 20 Batch 400 Loss 0.3554
Epoch 20 Batch 500 Loss 0.2905
Epoch 20 Batch 600 Loss 0.2984
Epoch 20 Batch 700 Loss 0.3065
Epoch 20 Batch 800 Loss 0.3337
Epoch 20 Batch 900 Loss 0.2904
Epoch 20 Batch 1000 Loss 0.3287
Epoch 20 Batch 1100 Loss 0.3136
Epoch 20 Batch 1200 Loss 0.3474
Epoch 20 Loss 0.316204
Time taken for 1 epoch 176.05854892730713 sec

No description has been provided for this image
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder_gru_resnet, decoder_gru_resnet)
    
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
display_bleu_score(image, result)
./train2014/COCO_train2014_000000461189.jpg
Real Caption: <start> a kitchen with an oven and a table <end>
Prediction Caption: a view of a large kitchen with wooden cabinets and a wooden floor and black countertops counter space and white cabinets <end>
No description has been provided for this image
No description has been provided for this image
************************************************************
Predicted Caption :
a view of a large kitchen with wooden cabinets and a wooden floor and black countertops counter space and white cabinets <end>

************************************************************
References :
<start> a kitchen with an oven and a table <end>
<start> a kitchen with <unk> colored cupboards and a silver oven <end>
<start> the interior of a kitchen with multiple lighting and wood floors <end>
<start> a dimly lit kitchen with a modern dark decor <end>
<start> a large kitchen with black cabinets and silver stove <end>
...

************************************************************
BLEU Score :
unigram  = 0.4545454545
bigram   = 0.3603749851
trigram  = 0.2384833484
4-gram = 0.1616921435
************************************************************
In [ ]:
encoder_gru_resnet.save('models/captioning_models/encoder_resnet_gru_model.keras')
decoder_gru_resnet.save('models/captioning_models/decoder_resnet_gru_model.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_resnet, decoder_gru_resnet, img_name_val, cap_val, tokenizer)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 2-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 3-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 4-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.1034515997
bigram   = 0.0000000000
trigram  = 0.0000000000
4-gram = 0.0000000000
************************************************************
(0.10345159969758377,
 4.439541344333561e-155,
 3.857526580399094e-204,
 9.458452165801623e-232)

Model 3¶

3 Layers GRU with BLEU and ResNet50

In [ ]:
# Instantiate encoder and decoder
encoder_gru_L3_resnet, decoder_gru_L3_resnet = create_model(CNN_Encoder, RNN_Decoder_GRU_3L, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_L3_resnet, 
                           decoder=decoder_gru_L3_resnet, 
                           optimizer=optimizer)
checkpoint_path = "./checkpoints/gru-l3-attention"
ckpt_manager = tf.train.CheckpointManager(ckpt, 
                                          checkpoint_path, 
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_L3_resnet, decoder_gru_L3_resnet, ckpt_manager)
Epoch 1 Batch 0 Loss 1.8800
Epoch 1 Batch 100 Loss 1.3634
Epoch 1 Batch 200 Loss 1.2763
Epoch 1 Batch 300 Loss 1.2612
Epoch 1 Batch 400 Loss 1.3582
Epoch 1 Batch 500 Loss 1.4150
Epoch 1 Batch 600 Loss 1.1775
Epoch 1 Batch 700 Loss 1.3302
Epoch 1 Batch 800 Loss 1.0843
Epoch 1 Batch 900 Loss 1.1823
Epoch 1 Batch 1000 Loss 1.2612
Epoch 1 Batch 1100 Loss 1.4069
Epoch 1 Batch 1200 Loss 1.3640
2024-10-23 21:08:49.586463: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 1.275946
Time taken for 1 epoch 392.353985786438 sec

Epoch 2 Batch 0 Loss 1.4340
Epoch 2 Batch 100 Loss 1.4314
Epoch 2 Batch 200 Loss 1.0692
Epoch 2 Batch 300 Loss 1.0366
Epoch 2 Batch 400 Loss 1.0671
Epoch 2 Batch 500 Loss 0.9356
Epoch 2 Batch 600 Loss 0.9546
Epoch 2 Batch 700 Loss 0.9686
Epoch 2 Batch 800 Loss 0.9334
Epoch 2 Batch 900 Loss 0.8734
Epoch 2 Batch 1000 Loss 0.9508
Epoch 2 Batch 1100 Loss 0.9545
Epoch 2 Batch 1200 Loss 0.9678
Epoch 2 Loss 1.018137
Time taken for 1 epoch 261.60807394981384 sec

Epoch 3 Batch 0 Loss 0.8519
Epoch 3 Batch 100 Loss 0.9725
Epoch 3 Batch 200 Loss 0.7563
Epoch 3 Batch 300 Loss 0.9703
Epoch 3 Batch 400 Loss 0.9373
Epoch 3 Batch 500 Loss 0.8537
Epoch 3 Batch 600 Loss 0.8569
Epoch 3 Batch 700 Loss 0.9496
Epoch 3 Batch 800 Loss 1.0092
Epoch 3 Batch 900 Loss 0.7985
Epoch 3 Batch 1000 Loss 0.8884
Epoch 3 Batch 1100 Loss 0.7937
Epoch 3 Batch 1200 Loss 0.9104
2024-10-23 21:17:34.670808: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.863571
Time taken for 1 epoch 263.3019468784332 sec

Epoch 4 Batch 0 Loss 0.7813
Epoch 4 Batch 100 Loss 0.8211
Epoch 4 Batch 200 Loss 0.7291
Epoch 4 Batch 300 Loss 0.6982
Epoch 4 Batch 400 Loss 0.9262
Epoch 4 Batch 500 Loss 0.7782
Epoch 4 Batch 600 Loss 0.7966
Epoch 4 Batch 700 Loss 0.9356
Epoch 4 Batch 800 Loss 0.7733
Epoch 4 Batch 900 Loss 0.7418
Epoch 4 Batch 1000 Loss 0.8871
Epoch 4 Batch 1100 Loss 0.7817
Epoch 4 Batch 1200 Loss 0.6912
Epoch 4 Loss 0.806589
Time taken for 1 epoch 264.0402846336365 sec

Epoch 5 Batch 0 Loss 0.8809
Epoch 5 Batch 100 Loss 0.7652
Epoch 5 Batch 200 Loss 0.8330
Epoch 5 Batch 300 Loss 0.8552
Epoch 5 Batch 400 Loss 0.8630
Epoch 5 Batch 500 Loss 0.9165
Epoch 5 Batch 600 Loss 0.6650
Epoch 5 Batch 700 Loss 0.9382
Epoch 5 Batch 800 Loss 0.7748
Epoch 5 Batch 900 Loss 0.7450
Epoch 5 Batch 1000 Loss 0.7221
Epoch 5 Batch 1100 Loss 0.6803
Epoch 5 Batch 1200 Loss 0.8076
Epoch 5 Loss 0.769270
Time taken for 1 epoch 263.58511996269226 sec

Epoch 6 Batch 0 Loss 0.9396
Epoch 6 Batch 100 Loss 0.7709
Epoch 6 Batch 200 Loss 0.7401
Epoch 6 Batch 300 Loss 0.8272
Epoch 6 Batch 400 Loss 0.6988
Epoch 6 Batch 500 Loss 0.6924
Epoch 6 Batch 600 Loss 0.6814
Epoch 6 Batch 700 Loss 0.8023
Epoch 6 Batch 800 Loss 0.6797
Epoch 6 Batch 900 Loss 0.6853
Epoch 6 Batch 1000 Loss 0.6215
Epoch 6 Batch 1100 Loss 0.7544
Epoch 6 Batch 1200 Loss 0.7260
Epoch 6 Loss 0.738621
Time taken for 1 epoch 262.6053535938263 sec

Epoch 7 Batch 0 Loss 0.7672
Epoch 7 Batch 100 Loss 0.7677
Epoch 7 Batch 200 Loss 0.6071
Epoch 7 Batch 300 Loss 0.7035
Epoch 7 Batch 400 Loss 0.6916
Epoch 7 Batch 500 Loss 0.6411
Epoch 7 Batch 600 Loss 0.7217
Epoch 7 Batch 700 Loss 0.7668
Epoch 7 Batch 800 Loss 0.7479
Epoch 7 Batch 900 Loss 0.7643
Epoch 7 Batch 1000 Loss 0.8137
Epoch 7 Batch 1100 Loss 0.7446
Epoch 7 Batch 1200 Loss 0.6719
2024-10-23 21:35:07.461676: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.712677
Time taken for 1 epoch 262.56003999710083 sec

Epoch 8 Batch 0 Loss 0.8302
Epoch 8 Batch 100 Loss 0.7190
Epoch 8 Batch 200 Loss 0.6850
Epoch 8 Batch 300 Loss 0.6612
Epoch 8 Batch 400 Loss 0.6782
Epoch 8 Batch 500 Loss 0.7693
Epoch 8 Batch 600 Loss 0.6354
Epoch 8 Batch 700 Loss 0.7167
Epoch 8 Batch 800 Loss 0.6314
Epoch 8 Batch 900 Loss 0.7537
Epoch 8 Batch 1000 Loss 0.6188
Epoch 8 Batch 1100 Loss 0.6421
Epoch 8 Batch 1200 Loss 0.6549
Epoch 8 Loss 0.688417
Time taken for 1 epoch 263.40365076065063 sec

Epoch 9 Batch 0 Loss 0.6985
Epoch 9 Batch 100 Loss 0.7271
Epoch 9 Batch 200 Loss 0.5734
Epoch 9 Batch 300 Loss 0.5354
Epoch 9 Batch 400 Loss 0.6367
Epoch 9 Batch 500 Loss 0.6057
Epoch 9 Batch 600 Loss 0.5762
Epoch 9 Batch 700 Loss 0.6228
Epoch 9 Batch 800 Loss 0.6960
Epoch 9 Batch 900 Loss 0.7059
Epoch 9 Batch 1000 Loss 0.7739
Epoch 9 Batch 1100 Loss 0.6428
Epoch 9 Batch 1200 Loss 0.7124
Epoch 9 Loss 0.666510
Time taken for 1 epoch 261.74695348739624 sec

Epoch 10 Batch 0 Loss 0.7298
Epoch 10 Batch 100 Loss 0.7012
Epoch 10 Batch 200 Loss 0.5880
Epoch 10 Batch 300 Loss 0.5870
Epoch 10 Batch 400 Loss 0.7663
Epoch 10 Batch 500 Loss 0.5953
Epoch 10 Batch 600 Loss 0.6331
Epoch 10 Batch 700 Loss 0.6985
Epoch 10 Batch 800 Loss 0.5709
Epoch 10 Batch 900 Loss 0.6214
Epoch 10 Batch 1000 Loss 0.5542
Epoch 10 Batch 1100 Loss 0.6052
Epoch 10 Batch 1200 Loss 0.6134
Epoch 10 Loss 0.646228
Time taken for 1 epoch 261.6729748249054 sec

Epoch 11 Batch 0 Loss 0.7387
Epoch 11 Batch 100 Loss 0.6400
Epoch 11 Batch 200 Loss 0.6309
Epoch 11 Batch 300 Loss 0.6331
Epoch 11 Batch 400 Loss 0.6732
Epoch 11 Batch 500 Loss 0.6432
Epoch 11 Batch 600 Loss 0.5667
Epoch 11 Batch 700 Loss 0.6931
Epoch 11 Batch 800 Loss 0.4984
Epoch 11 Batch 900 Loss 0.6606
Epoch 11 Batch 1000 Loss 0.6421
Epoch 11 Batch 1100 Loss 0.7174
Epoch 11 Batch 1200 Loss 0.6267
Epoch 11 Loss 0.626057
Time taken for 1 epoch 261.69165444374084 sec

Epoch 12 Batch 0 Loss 0.5977
Epoch 12 Batch 100 Loss 0.5946
Epoch 12 Batch 200 Loss 0.5669
Epoch 12 Batch 300 Loss 0.6404
Epoch 12 Batch 400 Loss 0.6047
Epoch 12 Batch 500 Loss 0.5737
Epoch 12 Batch 600 Loss 0.6484
Epoch 12 Batch 700 Loss 0.6829
Epoch 12 Batch 800 Loss 0.5373
Epoch 12 Batch 900 Loss 0.5781
Epoch 12 Batch 1000 Loss 0.5809
Epoch 12 Batch 1100 Loss 0.6197
Epoch 12 Batch 1200 Loss 0.5552
Epoch 12 Loss 0.607003
Time taken for 1 epoch 262.31527185440063 sec

Epoch 13 Batch 0 Loss 0.6487
Epoch 13 Batch 100 Loss 0.6125
Epoch 13 Batch 200 Loss 0.5749
Epoch 13 Batch 300 Loss 0.6284
Epoch 13 Batch 400 Loss 0.5512
Epoch 13 Batch 500 Loss 0.5530
Epoch 13 Batch 600 Loss 0.5457
Epoch 13 Batch 700 Loss 0.5863
Epoch 13 Batch 800 Loss 0.5973
Epoch 13 Batch 900 Loss 0.6671
Epoch 13 Batch 1000 Loss 0.6760
Epoch 13 Batch 1100 Loss 0.5488
Epoch 13 Batch 1200 Loss 0.6886
Epoch 13 Loss 0.587778
Time taken for 1 epoch 263.3172333240509 sec

Epoch 14 Batch 0 Loss 0.5918
Epoch 14 Batch 100 Loss 0.6638
Epoch 14 Batch 200 Loss 0.5363
Epoch 14 Batch 300 Loss 0.5382
Epoch 14 Batch 400 Loss 0.6229
Epoch 14 Batch 500 Loss 0.5044
Epoch 14 Batch 600 Loss 0.5102
Epoch 14 Batch 700 Loss 0.6639
Epoch 14 Batch 800 Loss 0.5602
Epoch 14 Batch 900 Loss 0.5231
Epoch 14 Batch 1000 Loss 0.5534
Epoch 14 Batch 1100 Loss 0.5675
Epoch 14 Batch 1200 Loss 0.5104
Epoch 14 Loss 0.569652
Time taken for 1 epoch 262.9858510494232 sec

Epoch 15 Batch 0 Loss 0.6006
Epoch 15 Batch 100 Loss 0.5741
Epoch 15 Batch 200 Loss 0.5694
Epoch 15 Batch 300 Loss 0.5299
Epoch 15 Batch 400 Loss 0.6709
Epoch 15 Batch 500 Loss 0.5226
Epoch 15 Batch 600 Loss 0.5265
Epoch 15 Batch 700 Loss 0.7045
Epoch 15 Batch 800 Loss 0.5379
Epoch 15 Batch 900 Loss 0.6596
Epoch 15 Batch 1000 Loss 0.6422
Epoch 15 Batch 1100 Loss 0.5319
Epoch 15 Batch 1200 Loss 0.5121
2024-10-23 22:10:08.272078: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.552805
Time taken for 1 epoch 263.6769394874573 sec

Epoch 16 Batch 0 Loss 0.5874
Epoch 16 Batch 100 Loss 0.5352
Epoch 16 Batch 200 Loss 0.6396
Epoch 16 Batch 300 Loss 0.5557
Epoch 16 Batch 400 Loss 0.5130
Epoch 16 Batch 500 Loss 0.6142
Epoch 16 Batch 600 Loss 0.5125
Epoch 16 Batch 700 Loss 0.4839
Epoch 16 Batch 800 Loss 0.4985
Epoch 16 Batch 900 Loss 0.4384
Epoch 16 Batch 1000 Loss 0.5347
Epoch 16 Batch 1100 Loss 0.5782
Epoch 16 Batch 1200 Loss 0.4964
Epoch 16 Loss 0.536064
Time taken for 1 epoch 263.8788628578186 sec

Epoch 17 Batch 0 Loss 0.5677
Epoch 17 Batch 100 Loss 0.5209
Epoch 17 Batch 200 Loss 0.5020
Epoch 17 Batch 300 Loss 0.5294
Epoch 17 Batch 400 Loss 0.5991
Epoch 17 Batch 500 Loss 0.5233
Epoch 17 Batch 600 Loss 0.4001
Epoch 17 Batch 700 Loss 0.5331
Epoch 17 Batch 800 Loss 0.4739
Epoch 17 Batch 900 Loss 0.4897
Epoch 17 Batch 1000 Loss 0.5354
Epoch 17 Batch 1100 Loss 0.4603
Epoch 17 Batch 1200 Loss 0.4435
Epoch 17 Loss 0.520028
Time taken for 1 epoch 264.11608695983887 sec

Epoch 18 Batch 0 Loss 0.5663
Epoch 18 Batch 100 Loss 0.6248
Epoch 18 Batch 200 Loss 0.5025
Epoch 18 Batch 300 Loss 0.5184
Epoch 18 Batch 400 Loss 0.4410
Epoch 18 Batch 500 Loss 0.4576
Epoch 18 Batch 600 Loss 0.5157
Epoch 18 Batch 700 Loss 0.5508
Epoch 18 Batch 800 Loss 0.4744
Epoch 18 Batch 900 Loss 0.4751
Epoch 18 Batch 1000 Loss 0.5086
Epoch 18 Batch 1100 Loss 0.4914
Epoch 18 Batch 1200 Loss 0.5353
Epoch 18 Loss 0.503900
Time taken for 1 epoch 263.3749485015869 sec

Epoch 19 Batch 0 Loss 0.4303
Epoch 19 Batch 100 Loss 0.5605
Epoch 19 Batch 200 Loss 0.4685
Epoch 19 Batch 300 Loss 0.4779
Epoch 19 Batch 400 Loss 0.5489
Epoch 19 Batch 500 Loss 0.4112
Epoch 19 Batch 600 Loss 0.4930
Epoch 19 Batch 700 Loss 0.5393
Epoch 19 Batch 800 Loss 0.4403
Epoch 19 Batch 900 Loss 0.5373
Epoch 19 Batch 1000 Loss 0.4607
Epoch 19 Batch 1100 Loss 0.5792
Epoch 19 Batch 1200 Loss 0.4879
Epoch 19 Loss 0.489598
Time taken for 1 epoch 263.7753279209137 sec

Epoch 20 Batch 0 Loss 0.4724
Epoch 20 Batch 100 Loss 0.3602
Epoch 20 Batch 200 Loss 0.4118
Epoch 20 Batch 300 Loss 0.5279
Epoch 20 Batch 400 Loss 0.5263
Epoch 20 Batch 500 Loss 0.5329
Epoch 20 Batch 600 Loss 0.5500
Epoch 20 Batch 700 Loss 0.4676
Epoch 20 Batch 800 Loss 0.4057
Epoch 20 Batch 900 Loss 0.4812
Epoch 20 Batch 1000 Loss 0.5685
Epoch 20 Batch 1100 Loss 0.4559
Epoch 20 Batch 1200 Loss 0.4333
Epoch 20 Loss 0.474137
Time taken for 1 epoch 263.65032482147217 sec

No description has been provided for this image
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder_gru_L3_resnet, decoder_gru_L3_resnet)
    
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
display_bleu_score(image, result)
./train2014/COCO_train2014_000000274277.jpg
W0000 00:00:1729715528.379348    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.380787    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.382240    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.383702    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.385150    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.386612    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.388053    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.389501    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.390958    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.392489    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.394050    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.395597    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.397332    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.399031    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.400794    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.402434    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.404337    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.442045    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.443380    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.444755    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.446136    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.447493    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.448877    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.450251    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.451620    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.452995    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.454347    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.455728    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.457082    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.458472    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.459857    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.461231    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.462638    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.463994    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.465399    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.466857    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.468260    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.469628    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.471003    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.472402    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.473859    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.475283    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.476774    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.478212    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.479661    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.481098    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.482513    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.483954    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.485378    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.486803    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.488384    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.489901    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.491446    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.492948    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.494392    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.495812    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.497243    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.550585    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.551931    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.553277    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.554635    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.556008    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.557361    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.558705    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.560081    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.561438    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.562850    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.564226    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.565586    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.566880    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.568253    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.569630    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.571065    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.572453    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.573842    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.575240    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.576615    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.577994    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.579364    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.580733    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.582055    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.583418    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.584712    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.586080    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.587447    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.588861    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.590236    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.591586    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.592962    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.594307    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.595737    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.597161    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.598631    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.599980    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.601457    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.602837    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.604179    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.651458    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.652783    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.654184    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.655568    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.656938    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.658296    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.659668    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.661019    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.662389    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.663760    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.665108    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.666462    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.667828    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.669175    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.670533    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.671881    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.673249    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.674647    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.676079    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.677436    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.678807    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.680249    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.681689    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.683233    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.684623    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.686102    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.687478    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.688918    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.690323    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.691730    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.693118    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.694518    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.695922    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.697358    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.698791    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.700271    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.701708    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.703175    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.704661    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.706182    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.742811    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.744317    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.745741    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.747164    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.748645    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.750107    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.751563    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.752989    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.754453    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.755867    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.757303    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.758789    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.760246    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.761681    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.763114    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.764683    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.766099    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.767561    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.769011    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.770475    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.771955    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.773410    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.774840    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.776294    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.777886    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.779345    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.780828    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.782327    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.783813    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.785225    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.786607    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.788080    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.789555    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.791081    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.792584    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.794057    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.795401    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.796873    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.798341    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.799849    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.857499    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.858889    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.860254    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.861645    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.863030    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.864402    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.865773    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.867117    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.868504    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.869876    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.871235    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.872627    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.873984    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.875383    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.876757    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.878162    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.879567    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.880929    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.882279    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.883670    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.885046    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.886413    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.887794    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.889209    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.890613    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.892000    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.893381    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.894759    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.896183    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.897586    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.898973    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.900275    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.901618    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.903012    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.904444    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.905898    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.907280    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.908740    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.910090    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.911453    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.946828    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.948158    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.949575    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.950961    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.952395    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.953800    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.955163    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.956532    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.957910    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.959385    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.960781    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.962216    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.963628    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.965030    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.966454    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.967863    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.969288    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.970714    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.972131    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.973575    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.975023    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.976517    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.978019    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.979486    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.980916    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.982425    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.983987    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.985504    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.987053    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.988559    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.990069    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.991593    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.993083    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.994614    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.996108    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.997630    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715528.999159    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.000741    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.002477    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.042991    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.044419    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.045957    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.047484    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.049067    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.050652    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.052238    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.053808    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.055357    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.056866    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.058400    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.059927    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.061568    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.063202    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.064738    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.066245    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.067857    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.069420    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.071005    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.072623    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.074162    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.075708    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.077340    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.078871    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.080263    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.081833    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.083361    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.084949    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.086488    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.088036    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.089627    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.091219    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.092561    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.094106    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.095720    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.097315    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.098898    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.100387    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.101801    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.103236    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.183029    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.184468    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.185985    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.187523    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.189081    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.190582    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.192105    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.193627    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.195172    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.196561    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.198091    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.199447    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.201069    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.202661    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.204247    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.205764    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.207321    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.208880    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.210431    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.211958    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.213601    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.215194    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.216789    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.218333    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.219699    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.221242    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.222834    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.224251    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.225812    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.227428    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.228968    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.230568    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.232136    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.233656    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.235047    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.236609    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.238043    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.239678    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.281088    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.282635    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.284260    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.285891    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.287451    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.289015    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.290702    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.292288    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.293962    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.295518    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.297077    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.298948    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.300643    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.302274    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.303974    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.305630    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.307312    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.308904    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.310899    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.312848    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.314548    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.316352    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.318017    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.319724    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.321414    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.323219    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.325029    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.326890    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.328725    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.330527    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.332270    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.334075    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.335756    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.337574    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.339239    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.341106    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.343274    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.385529    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.386993    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.388550    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.390072    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.391526    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.392996    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.394527    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.396075    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.397578    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.399093    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.401365    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.403556    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.405734    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.407837    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.409991    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.412113    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.414205    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.416383    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.417939    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.420041    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.422313    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.423852    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.425959    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.428072    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.430309    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.432576    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.434662    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.436756    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.438953    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.441058    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.443173    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.445343    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.447438    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.449562    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.451671    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.453809    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.455902    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.458106    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729715529.460312    9838 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
Real Caption: <start> a small child is petting a brown cow by a fence <end>
Prediction Caption: a boy is working on a keyboard in kitchen levitating in a blender <end>
No description has been provided for this image
No description has been provided for this image
************************************************************
Predicted Caption :
a boy is working on a keyboard in kitchen levitating in a blender <end>

************************************************************
References :
<start> a small child is petting a brown cow by a fence <end>
<start> a father holds his daughter and <unk> her pet a cow <end>
<start> a young girl is reaching out to pet a large cow <end>
<start> a man holding a baby who is touching a cow <end>
<start> the young child is petting the cow at the farm <end>
...

************************************************************
BLEU Score :
unigram  = 0.3571428571
bigram   = 0.0000000000
trigram  = 0.0000000000
4-gram = 0.0000000000
************************************************************
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 2-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 3-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 4-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
In [ ]:
encoder_gru_L3_resnet.save('models/captioning_models/encoder_resnet_gru_model.keras')
decoder_gru_L3_resnet.save('models/captioning_models/decoder_resnet_gru_model.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_L3_resnet, decoder_gru_L3_resnet, img_name_val, cap_val, tokenizer)
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.0972967400
bigram   = 0.0000000000
trigram  = 0.0000000000
4-gram = 0.0000000000
************************************************************
(0.09729673999997569,
 4.261733441016533e-155,
 3.7465316748112534e-204,
 9.244315369840666e-232)
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]

Model 4¶

LSTM with BLEU and ResNet50

In [ ]:
# Instantiate encoder and decoder
encoder_lstm, decoder_lstm = create_model(CNN_Encoder, RNN_Decoder_LSTM, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_lstm,
                           decoder=decoder_lstm,
                           optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
                                          checkpoint_path,
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_lstm, decoder_lstm, ckpt_manager)
/usr/local/lib/python3.10/dist-packages/keras/src/optimizers/base_optimizer.py:664: UserWarning: Gradients do not exist for variables ['kernel', 'kernel', 'kernel', 'bias', 'kernel'] when minimizing the loss. If using `model.compile()`, did you forget to provide a `loss` argument?
  warnings.warn(
Epoch 1 Batch 0 Loss 1.3831
Epoch 1 Batch 100 Loss 0.9723
Epoch 1 Batch 200 Loss 0.8904
Epoch 1 Batch 300 Loss 0.8751
Epoch 1 Batch 400 Loss 0.8336
Epoch 1 Batch 500 Loss 0.7832
Epoch 1 Batch 600 Loss 0.7403
Epoch 1 Loss 0.870570
Time taken for 1 epoch 152.00215911865234 sec

Epoch 2 Batch 0 Loss 0.8049
Epoch 2 Batch 100 Loss 0.8281
Epoch 2 Batch 200 Loss 0.7264
Epoch 2 Batch 300 Loss 0.6799
Epoch 2 Batch 400 Loss 0.6123
Epoch 2 Batch 500 Loss 0.7127
Epoch 2 Batch 600 Loss 0.6841
Epoch 2 Loss 0.738784
Time taken for 1 epoch 93.13927578926086 sec

Epoch 3 Batch 0 Loss 0.6805
Epoch 3 Batch 100 Loss 0.6375
Epoch 3 Batch 200 Loss 0.7164
Epoch 3 Batch 300 Loss 0.6625
Epoch 3 Batch 400 Loss 0.5675
Epoch 3 Batch 500 Loss 0.6545
Epoch 3 Batch 600 Loss 0.6307
Epoch 3 Loss 0.678108
Time taken for 1 epoch 93.11038064956665 sec

Epoch 4 Batch 0 Loss 0.7152
Epoch 4 Batch 100 Loss 0.6767
Epoch 4 Batch 200 Loss 0.6591
Epoch 4 Batch 300 Loss 0.5683
Epoch 4 Batch 400 Loss 0.6328
Epoch 4 Batch 500 Loss 0.6781
Epoch 4 Batch 600 Loss 0.5984
Epoch 4 Loss 0.633837
Time taken for 1 epoch 93.25185775756836 sec

Epoch 5 Batch 0 Loss 0.6055
Epoch 5 Batch 100 Loss 0.5480
Epoch 5 Batch 200 Loss 0.5922
Epoch 5 Batch 300 Loss 0.5487
Epoch 5 Batch 400 Loss 0.5750
Epoch 5 Batch 500 Loss 0.5540
Epoch 5 Batch 600 Loss 0.5647
Epoch 5 Loss 0.597418
Time taken for 1 epoch 92.65495610237122 sec

Epoch 6 Batch 0 Loss 0.5252
Epoch 6 Batch 100 Loss 0.5995
Epoch 6 Batch 200 Loss 0.5775
Epoch 6 Batch 300 Loss 0.5506
Epoch 6 Batch 400 Loss 0.5226
Epoch 6 Batch 500 Loss 0.5735
Epoch 6 Batch 600 Loss 0.5934
Epoch 6 Loss 0.565422
Time taken for 1 epoch 99.13107991218567 sec

Epoch 7 Batch 0 Loss 0.4687
Epoch 7 Batch 100 Loss 0.5263
Epoch 7 Batch 200 Loss 0.5055
Epoch 7 Batch 300 Loss 0.4717
Epoch 7 Batch 400 Loss 0.5540
Epoch 7 Batch 500 Loss 0.5021
Epoch 7 Batch 600 Loss 0.6039
Epoch 7 Loss 0.536423
Time taken for 1 epoch 93.5994622707367 sec

Epoch 8 Batch 0 Loss 0.5791
Epoch 8 Batch 100 Loss 0.5324
Epoch 8 Batch 200 Loss 0.5112
Epoch 8 Batch 300 Loss 0.5310
Epoch 8 Batch 400 Loss 0.5210
Epoch 8 Batch 500 Loss 0.4669
Epoch 8 Batch 600 Loss 0.4941
Epoch 8 Loss 0.509094
Time taken for 1 epoch 93.1858274936676 sec

Epoch 9 Batch 0 Loss 0.4531
Epoch 9 Batch 100 Loss 0.4827
Epoch 9 Batch 200 Loss 0.4385
Epoch 9 Batch 300 Loss 0.5299
Epoch 9 Batch 400 Loss 0.4694
Epoch 9 Batch 500 Loss 0.4675
Epoch 9 Batch 600 Loss 0.4686
Epoch 9 Loss 0.484041
Time taken for 1 epoch 93.07637643814087 sec

Epoch 10 Batch 0 Loss 0.4952
Epoch 10 Batch 100 Loss 0.4532
Epoch 10 Batch 200 Loss 0.4576
Epoch 10 Batch 300 Loss 0.4407
Epoch 10 Batch 400 Loss 0.4725
Epoch 10 Batch 500 Loss 0.4470
Epoch 10 Batch 600 Loss 0.4543
Epoch 10 Loss 0.460005
Time taken for 1 epoch 93.67944312095642 sec

Epoch 11 Batch 0 Loss 0.4750
Epoch 11 Batch 100 Loss 0.4594
Epoch 11 Batch 200 Loss 0.4497
Epoch 11 Batch 300 Loss 0.4565
Epoch 11 Batch 400 Loss 0.4069
Epoch 11 Batch 500 Loss 0.4643
Epoch 11 Batch 600 Loss 0.4308
Epoch 11 Loss 0.436256
Time taken for 1 epoch 99.09716939926147 sec

Epoch 12 Batch 0 Loss 0.4806
Epoch 12 Batch 100 Loss 0.4399
Epoch 12 Batch 200 Loss 0.4399
Epoch 12 Batch 300 Loss 0.4013
Epoch 12 Batch 400 Loss 0.3795
Epoch 12 Batch 500 Loss 0.4337
Epoch 12 Batch 600 Loss 0.3773
Epoch 12 Loss 0.415265
Time taken for 1 epoch 94.1980767250061 sec

Epoch 13 Batch 0 Loss 0.4185
Epoch 13 Batch 100 Loss 0.4050
Epoch 13 Batch 200 Loss 0.4159
Epoch 13 Batch 300 Loss 0.3988
Epoch 13 Batch 400 Loss 0.3430
Epoch 13 Batch 500 Loss 0.4026
Epoch 13 Batch 600 Loss 0.3719
Epoch 13 Loss 0.395274
Time taken for 1 epoch 108.24577379226685 sec

Epoch 14 Batch 0 Loss 0.3550
Epoch 14 Batch 100 Loss 0.3611
Epoch 14 Batch 200 Loss 0.3304
Epoch 14 Batch 300 Loss 0.3564
Epoch 14 Batch 400 Loss 0.3474
Epoch 14 Batch 500 Loss 0.3742
Epoch 14 Batch 600 Loss 0.3846
Epoch 14 Loss 0.375913
Time taken for 1 epoch 94.54929637908936 sec

Epoch 15 Batch 0 Loss 0.3404
Epoch 15 Batch 100 Loss 0.3659
Epoch 15 Batch 200 Loss 0.3987
Epoch 15 Batch 300 Loss 0.3594
Epoch 15 Batch 400 Loss 0.3763
Epoch 15 Batch 500 Loss 0.3899
Epoch 15 Batch 600 Loss 0.3747
Epoch 15 Loss 0.357651
Time taken for 1 epoch 94.3346164226532 sec

Epoch 16 Batch 0 Loss 0.3317
Epoch 16 Batch 100 Loss 0.3683
Epoch 16 Batch 200 Loss 0.3701
Epoch 16 Batch 300 Loss 0.3195
Epoch 16 Batch 400 Loss 0.3268
Epoch 16 Batch 500 Loss 0.3668
Epoch 16 Batch 600 Loss 0.2972
Epoch 16 Loss 0.341543
Time taken for 1 epoch 99.35520458221436 sec

Epoch 17 Batch 0 Loss 0.3200
Epoch 17 Batch 100 Loss 0.3107
Epoch 17 Batch 200 Loss 0.3493
Epoch 17 Batch 300 Loss 0.3187
Epoch 17 Batch 400 Loss 0.2750
Epoch 17 Batch 500 Loss 0.2709
Epoch 17 Batch 600 Loss 0.3122
Epoch 17 Loss 0.326561
Time taken for 1 epoch 93.22835731506348 sec

Epoch 18 Batch 0 Loss 0.2596
Epoch 18 Batch 100 Loss 0.3018
Epoch 18 Batch 200 Loss 0.3063
Epoch 18 Batch 300 Loss 0.3251
Epoch 18 Batch 400 Loss 0.3383
Epoch 18 Batch 500 Loss 0.3007
Epoch 18 Batch 600 Loss 0.2908
Epoch 18 Loss 0.311323
Time taken for 1 epoch 93.99069356918335 sec

Epoch 19 Batch 0 Loss 0.2561
Epoch 19 Batch 100 Loss 0.2828
Epoch 19 Batch 200 Loss 0.2815
Epoch 19 Batch 300 Loss 0.2784
Epoch 19 Batch 400 Loss 0.2687
Epoch 19 Batch 500 Loss 0.2915
Epoch 19 Batch 600 Loss 0.2500
Epoch 19 Loss 0.298024
Time taken for 1 epoch 93.09543180465698 sec

Epoch 20 Batch 0 Loss 0.2711
Epoch 20 Batch 100 Loss 0.2958
Epoch 20 Batch 200 Loss 0.2883
Epoch 20 Batch 300 Loss 0.2930
Epoch 20 Batch 400 Loss 0.2953
Epoch 20 Batch 500 Loss 0.2748
Epoch 20 Batch 600 Loss 0.2866
Epoch 20 Loss 0.286795
Time taken for 1 epoch 92.65783095359802 sec

No description has been provided for this image
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder, decoder)

print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
transfer_learning/train2014/COCO_train2014_000000251219.jpg
Real Caption: <start> a kitchen with a refrigerator sink and shelving with various items <end>
Prediction Caption: a kitchen with a stove sink tissue and cabinets and stove and appliances <end>
No description has been provided for this image
In [ ]:
display_bleu_score(image, result)
No description has been provided for this image
************************************************************
Predicted Caption :
a kitchen with a stove sink tissue and cabinets and stove and appliances <end>

************************************************************
References :
<start> a white refrigerator freezer sitting in a kitchen <end>
<start> a white refrigerator that is in a kitchen <end>
<start> a photo of household kitchen with a traditional look <end>
<start> a modest kitchen with wooden cabinets and a spice shelf <end>
<start> a kitchen with a refrigerator sink and shelving with various items <end>
...

************************************************************
BLEU Score :
unigram  = 0.5714285714
bigram   = 0.4193139347
trigram  = 0.3119507414
4-gram = 0.2271870978
************************************************************
In [ ]:
# Save the Encoder model
encoder.save('models/captioning_models/encoder_lstm_resnet.keras')

# Save the Decoder model
decoder.save('models/captioning_models/decoder_lstm_resnet.keras')
In [ ]:
evaluate_average_bleu(encoder, decoder, img_name_val, cap_val, tokenizer)
/usr/local/lib/python3.10/dist-packages/nltk/translate/bleu_score.py:552: UserWarning: 
The hypothesis contains 0 counts of 2-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/usr/local/lib/python3.10/dist-packages/nltk/translate/bleu_score.py:552: UserWarning: 
The hypothesis contains 0 counts of 3-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/usr/local/lib/python3.10/dist-packages/nltk/translate/bleu_score.py:552: UserWarning: 
The hypothesis contains 0 counts of 4-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.1137287452
bigram   = 0.0000209531
trigram  = 0.0000000000
4-gram = 0.0000000000
************************************************************
(0.11372874523877911,
 2.0953124859522655e-05,
 1.345579927627796e-106,
 9.653423620941904e-159)

Model 5¶

3 Layers GRU with BLEU and InceptionV3

In [ ]:
# Instantiate encoder and decoder
encoder_gru_L3_incv3, decoder_gru_L3_incv3 = create_model(CNN_Encoder, RNN_Decoder_GRU_3L, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_L3_incv3, 
                           decoder=decoder_gru_L3_incv3, 
                           optimizer=optimizer)
checkpoint_path = "./checkpoints/gru-l3-incv3"
ckpt_manager = tf.train.CheckpointManager(ckpt, 
                                          checkpoint_path, 
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_L3_incv3, decoder_gru_L3_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 2.0273
Epoch 1 Batch 100 Loss 1.4244
Epoch 1 Batch 200 Loss 1.0996
Epoch 1 Batch 300 Loss 1.0915
Epoch 1 Batch 400 Loss 1.0891
Epoch 1 Batch 500 Loss 1.0066
Epoch 1 Batch 600 Loss 0.9345
Epoch 1 Batch 700 Loss 0.9635
Epoch 1 Batch 800 Loss 0.9110
Epoch 1 Batch 900 Loss 0.7981
Epoch 1 Batch 1000 Loss 0.8036
Epoch 1 Batch 1100 Loss 0.8040
Epoch 1 Batch 1200 Loss 0.8165
2024-10-24 10:06:07.386889: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 1 Loss 0.992230
Time taken for 1 epoch 403.0873613357544 sec

Epoch 2 Batch 0 Loss 0.8655
Epoch 2 Batch 100 Loss 0.8208
Epoch 2 Batch 200 Loss 0.8260
Epoch 2 Batch 300 Loss 0.8338
Epoch 2 Batch 400 Loss 0.9222
Epoch 2 Batch 500 Loss 0.7434
Epoch 2 Batch 600 Loss 0.8859
Epoch 2 Batch 700 Loss 0.7581
Epoch 2 Batch 800 Loss 0.7558
Epoch 2 Batch 900 Loss 0.9581
Epoch 2 Batch 1000 Loss 0.7948
Epoch 2 Batch 1100 Loss 0.6928
Epoch 2 Batch 1200 Loss 0.6924
Epoch 2 Loss 0.818949
Time taken for 1 epoch 269.3762722015381 sec

Epoch 3 Batch 0 Loss 0.8814
Epoch 3 Batch 100 Loss 0.7610
Epoch 3 Batch 200 Loss 0.7679
Epoch 3 Batch 300 Loss 0.7000
Epoch 3 Batch 400 Loss 0.7909
Epoch 3 Batch 500 Loss 0.8011
Epoch 3 Batch 600 Loss 0.8097
Epoch 3 Batch 700 Loss 0.9529
Epoch 3 Batch 800 Loss 0.7974
Epoch 3 Batch 900 Loss 0.8413
Epoch 3 Batch 1000 Loss 0.6295
Epoch 3 Batch 1100 Loss 0.6649
Epoch 3 Batch 1200 Loss 0.7107
2024-10-24 10:15:05.612939: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 3 Loss 0.763228
Time taken for 1 epoch 268.65580010414124 sec

Epoch 4 Batch 0 Loss 0.8183
Epoch 4 Batch 100 Loss 0.7115
Epoch 4 Batch 200 Loss 0.5065
Epoch 4 Batch 300 Loss 0.7208
Epoch 4 Batch 400 Loss 0.7995
Epoch 4 Batch 500 Loss 0.8565
Epoch 4 Batch 600 Loss 0.6849
Epoch 4 Batch 700 Loss 0.7132
Epoch 4 Batch 800 Loss 0.6334
Epoch 4 Batch 900 Loss 0.6796
Epoch 4 Batch 1000 Loss 0.7184
Epoch 4 Batch 1100 Loss 0.6818
Epoch 4 Batch 1200 Loss 0.6700
Epoch 4 Loss 0.727730
Time taken for 1 epoch 269.0176486968994 sec

Epoch 5 Batch 0 Loss 0.7593
Epoch 5 Batch 100 Loss 0.7043
Epoch 5 Batch 200 Loss 0.6963
Epoch 5 Batch 300 Loss 0.7011
Epoch 5 Batch 400 Loss 0.7265
Epoch 5 Batch 500 Loss 0.6103
Epoch 5 Batch 600 Loss 0.8109
Epoch 5 Batch 700 Loss 0.6947
Epoch 5 Batch 800 Loss 0.6715
Epoch 5 Batch 900 Loss 0.5703
Epoch 5 Batch 1000 Loss 0.6739
Epoch 5 Batch 1100 Loss 0.5613
Epoch 5 Batch 1200 Loss 0.5914
Epoch 5 Loss 0.700923
Time taken for 1 epoch 270.2045750617981 sec

Epoch 6 Batch 0 Loss 0.6572
Epoch 6 Batch 100 Loss 0.5697
Epoch 6 Batch 200 Loss 0.7420
Epoch 6 Batch 300 Loss 0.6157
Epoch 6 Batch 400 Loss 0.9177
Epoch 6 Batch 500 Loss 0.6105
Epoch 6 Batch 600 Loss 0.7113
Epoch 6 Batch 700 Loss 0.6540
Epoch 6 Batch 800 Loss 0.5586
Epoch 6 Batch 900 Loss 0.7040
Epoch 6 Batch 1000 Loss 0.6584
Epoch 6 Batch 1100 Loss 0.6369
Epoch 6 Batch 1200 Loss 0.6958
Epoch 6 Loss 0.673867
Time taken for 1 epoch 270.36888003349304 sec

Epoch 7 Batch 0 Loss 0.6811
Epoch 7 Batch 100 Loss 0.6239
Epoch 7 Batch 200 Loss 0.7708
Epoch 7 Batch 300 Loss 0.6421
Epoch 7 Batch 400 Loss 0.6111
Epoch 7 Batch 500 Loss 0.6117
Epoch 7 Batch 600 Loss 0.6681
Epoch 7 Batch 700 Loss 0.6419
Epoch 7 Batch 800 Loss 0.5460
Epoch 7 Batch 900 Loss 0.5885
Epoch 7 Batch 1000 Loss 0.6350
Epoch 7 Batch 1100 Loss 0.6750
Epoch 7 Batch 1200 Loss 0.6399
2024-10-24 10:33:03.323824: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 7 Loss 0.650628
Time taken for 1 epoch 268.11972188949585 sec

Epoch 8 Batch 0 Loss 0.6282
Epoch 8 Batch 100 Loss 0.6848
Epoch 8 Batch 200 Loss 0.6740
Epoch 8 Batch 300 Loss 0.5582
Epoch 8 Batch 400 Loss 0.5874
Epoch 8 Batch 500 Loss 0.6132
Epoch 8 Batch 600 Loss 0.6987
Epoch 8 Batch 700 Loss 0.6518
Epoch 8 Batch 800 Loss 0.5698
Epoch 8 Batch 900 Loss 0.6209
Epoch 8 Batch 1000 Loss 0.6382
Epoch 8 Batch 1100 Loss 0.5934
Epoch 8 Batch 1200 Loss 0.4897
Epoch 8 Loss 0.625871
Time taken for 1 epoch 269.6277639865875 sec

Epoch 9 Batch 0 Loss 0.5534
Epoch 9 Batch 100 Loss 0.6635
Epoch 9 Batch 200 Loss 0.7824
Epoch 9 Batch 300 Loss 0.7120
Epoch 9 Batch 400 Loss 0.5911
Epoch 9 Batch 500 Loss 0.5887
Epoch 9 Batch 600 Loss 0.5511
Epoch 9 Batch 700 Loss 0.5679
Epoch 9 Batch 800 Loss 0.4577
Epoch 9 Batch 900 Loss 0.5216
Epoch 9 Batch 1000 Loss 0.5345
Epoch 9 Batch 1100 Loss 0.5544
Epoch 9 Batch 1200 Loss 0.5633
Epoch 9 Loss 0.606553
Time taken for 1 epoch 269.325320482254 sec

Epoch 10 Batch 0 Loss 0.6784
Epoch 10 Batch 100 Loss 0.5170
Epoch 10 Batch 200 Loss 0.5869
Epoch 10 Batch 300 Loss 0.4897
Epoch 10 Batch 400 Loss 0.5863
Epoch 10 Batch 500 Loss 0.6526
Epoch 10 Batch 600 Loss 0.6119
Epoch 10 Batch 700 Loss 0.4459
Epoch 10 Batch 800 Loss 0.5375
Epoch 10 Batch 900 Loss 0.5255
Epoch 10 Batch 1000 Loss 0.5257
Epoch 10 Batch 1100 Loss 0.5891
Epoch 10 Batch 1200 Loss 0.5453
Epoch 10 Loss 0.587032
Time taken for 1 epoch 268.5027709007263 sec

Epoch 11 Batch 0 Loss 0.5894
Epoch 11 Batch 100 Loss 0.5438
Epoch 11 Batch 200 Loss 0.6820
Epoch 11 Batch 300 Loss 0.6612
Epoch 11 Batch 400 Loss 0.6664
Epoch 11 Batch 500 Loss 0.5198
Epoch 11 Batch 600 Loss 0.5490
Epoch 11 Batch 700 Loss 0.5411
Epoch 11 Batch 800 Loss 0.4953
Epoch 11 Batch 900 Loss 0.6317
Epoch 11 Batch 1000 Loss 0.5701
Epoch 11 Batch 1100 Loss 0.5690
Epoch 11 Batch 1200 Loss 0.5792
Epoch 11 Loss 0.565977
Time taken for 1 epoch 268.39085698127747 sec

Epoch 12 Batch 0 Loss 0.4760
Epoch 12 Batch 100 Loss 0.5237
Epoch 12 Batch 200 Loss 0.5141
Epoch 12 Batch 300 Loss 0.5526
Epoch 12 Batch 400 Loss 0.5591
Epoch 12 Batch 500 Loss 0.7082
Epoch 12 Batch 600 Loss 0.6175
Epoch 12 Batch 700 Loss 0.5937
Epoch 12 Batch 800 Loss 0.4652
Epoch 12 Batch 900 Loss 0.5401
Epoch 12 Batch 1000 Loss 0.5784
Epoch 12 Batch 1100 Loss 0.4975
Epoch 12 Batch 1200 Loss 0.5906
Epoch 12 Loss 0.543929
Time taken for 1 epoch 268.069402217865 sec

Epoch 13 Batch 0 Loss 0.5277
Epoch 13 Batch 100 Loss 0.5138
Epoch 13 Batch 200 Loss 0.4930
Epoch 13 Batch 300 Loss 0.5789
Epoch 13 Batch 400 Loss 0.4957
Epoch 13 Batch 500 Loss 0.5294
Epoch 13 Batch 600 Loss 0.5152
Epoch 13 Batch 700 Loss 0.5142
Epoch 13 Batch 800 Loss 0.5069
Epoch 13 Batch 900 Loss 0.5006
Epoch 13 Batch 1000 Loss 0.4732
Epoch 13 Batch 1100 Loss 0.4990
Epoch 13 Batch 1200 Loss 0.4456
Epoch 13 Loss 0.523744
Time taken for 1 epoch 268.15740752220154 sec

Epoch 14 Batch 0 Loss 0.5118
Epoch 14 Batch 100 Loss 0.5061
Epoch 14 Batch 200 Loss 0.5002
Epoch 14 Batch 300 Loss 0.4504
Epoch 14 Batch 400 Loss 0.4795
Epoch 14 Batch 500 Loss 0.5499
Epoch 14 Batch 600 Loss 0.5171
Epoch 14 Batch 700 Loss 0.4518
Epoch 14 Batch 800 Loss 0.4520
Epoch 14 Batch 900 Loss 0.5086
Epoch 14 Batch 1000 Loss 0.4989
Epoch 14 Batch 1100 Loss 0.4530
Epoch 14 Batch 1200 Loss 0.3852
Epoch 14 Loss 0.504677
Time taken for 1 epoch 267.6439895629883 sec

Epoch 15 Batch 0 Loss 0.5873
Epoch 15 Batch 100 Loss 0.5372
Epoch 15 Batch 200 Loss 0.5173
Epoch 15 Batch 300 Loss 0.5251
Epoch 15 Batch 400 Loss 0.5412
Epoch 15 Batch 500 Loss 0.4160
Epoch 15 Batch 600 Loss 0.4662
Epoch 15 Batch 700 Loss 0.4873
Epoch 15 Batch 800 Loss 0.4345
Epoch 15 Batch 900 Loss 0.5547
Epoch 15 Batch 1000 Loss 0.5292
Epoch 15 Batch 1100 Loss 0.5533
Epoch 15 Batch 1200 Loss 0.4562
2024-10-24 11:08:51.620486: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Epoch 15 Loss 0.484940
Time taken for 1 epoch 268.5790066719055 sec

Epoch 16 Batch 0 Loss 0.4544
Epoch 16 Batch 100 Loss 0.4405
Epoch 16 Batch 200 Loss 0.5063
Epoch 16 Batch 300 Loss 0.4461
Epoch 16 Batch 400 Loss 0.4816
Epoch 16 Batch 500 Loss 0.4430
Epoch 16 Batch 600 Loss 0.4726
Epoch 16 Batch 700 Loss 0.4879
Epoch 16 Batch 800 Loss 0.4485
Epoch 16 Batch 900 Loss 0.4748
Epoch 16 Batch 1000 Loss 0.4308
Epoch 16 Batch 1100 Loss 0.5551
Epoch 16 Batch 1200 Loss 0.4556
Epoch 16 Loss 0.472030
Time taken for 1 epoch 268.04731154441833 sec

Epoch 17 Batch 0 Loss 0.4523
Epoch 17 Batch 100 Loss 0.4665
Epoch 17 Batch 200 Loss 0.5285
Epoch 17 Batch 300 Loss 0.4544
Epoch 17 Batch 400 Loss 0.4492
Epoch 17 Batch 500 Loss 0.5061
Epoch 17 Batch 600 Loss 0.4172
Epoch 17 Batch 700 Loss 0.5060
Epoch 17 Batch 800 Loss 0.4197
Epoch 17 Batch 900 Loss 0.4326
Epoch 17 Batch 1000 Loss 0.4764
Epoch 17 Batch 1100 Loss 0.4239
Epoch 17 Batch 1200 Loss 0.4864
Epoch 17 Loss 0.453266
Time taken for 1 epoch 267.4763150215149 sec

Epoch 18 Batch 0 Loss 0.4541
Epoch 18 Batch 100 Loss 0.3888
Epoch 18 Batch 200 Loss 0.4631
Epoch 18 Batch 300 Loss 0.3991
Epoch 18 Batch 400 Loss 0.5477
Epoch 18 Batch 500 Loss 0.5158
Epoch 18 Batch 600 Loss 0.4004
Epoch 18 Batch 700 Loss 0.3718
Epoch 18 Batch 800 Loss 0.4610
Epoch 18 Batch 900 Loss 0.4034
Epoch 18 Batch 1000 Loss 0.4937
Epoch 18 Batch 1100 Loss 0.3711
Epoch 18 Batch 1200 Loss 0.3866
Epoch 18 Loss 0.444814
Time taken for 1 epoch 267.9072439670563 sec

Epoch 19 Batch 0 Loss 0.4711
Epoch 19 Batch 100 Loss 0.3697
Epoch 19 Batch 200 Loss 0.4379
Epoch 19 Batch 300 Loss 0.3641
Epoch 19 Batch 400 Loss 0.4565
Epoch 19 Batch 500 Loss 0.4404
Epoch 19 Batch 600 Loss 0.4251
Epoch 19 Batch 700 Loss 0.4838
Epoch 19 Batch 800 Loss 0.4342
Epoch 19 Batch 900 Loss 0.4219
Epoch 19 Batch 1000 Loss 0.4346
Epoch 19 Batch 1100 Loss 0.4481
Epoch 19 Batch 1200 Loss 0.4086
Epoch 19 Loss 0.416508
Time taken for 1 epoch 268.13540625572205 sec

Epoch 20 Batch 0 Loss 0.4549
Epoch 20 Batch 100 Loss 0.4114
Epoch 20 Batch 200 Loss 0.4560
Epoch 20 Batch 300 Loss 0.4523
Epoch 20 Batch 400 Loss 0.3907
Epoch 20 Batch 500 Loss 0.3577
Epoch 20 Batch 600 Loss 0.4230
Epoch 20 Batch 700 Loss 0.4810
Epoch 20 Batch 800 Loss 0.4282
Epoch 20 Batch 900 Loss 0.4475
Epoch 20 Batch 1000 Loss 0.3753
Epoch 20 Batch 1100 Loss 0.4372
Epoch 20 Batch 1200 Loss 0.4181
Epoch 20 Loss 0.404555
Time taken for 1 epoch 268.7165153026581 sec

No description has been provided for this image
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder_gru_L3_incv3, decoder_gru_L3_incv3)
    
print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
display_bleu_score(image, result)
./train2014/COCO_train2014_000000208956.jpg
W0000 00:00:1729762273.326213    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.331408    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.332728    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.334061    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.335404    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.336757    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.338166    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.339525    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.340884    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.342257    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.344827    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.346301    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.347922    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.349417    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.350877    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.352382    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.353856    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.355350    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.356892    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.358480    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.359998    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.361616    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.364751    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.366810    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.368367    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.369962    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.371487    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.373134    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.374740    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.376296    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.414557    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.415975    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.417455    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.418919    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.420397    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.421949    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.423452    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.424957    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.426545    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.428085    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.429882    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.431476    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.433093    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.434716    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.436257    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.437831    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.439446    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.441079    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.442681    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.444349    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.445996    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.447674    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.449217    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.450846    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.452482    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.454020    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.455577    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.457307    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.459043    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.460759    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.462495    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.464181    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.465975    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.467668    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.469421    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.471264    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.473215    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.475287    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.511177    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.512683    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.514173    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.515681    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.517176    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.518674    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.520212    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.521802    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.523411    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.524997    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.526667    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.528298    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.529910    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.531548    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.533212    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.534837    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.536463    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.538220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.539915    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.541751    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.543409    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.545109    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.546862    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.548766    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.550582    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.552400    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.554240    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.555872    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.557515    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.559466    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.561316    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.563061    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.564949    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.566574    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.569902    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.572044    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.573975    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.576223    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.613888    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.615472    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.617098    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.618702    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.620329    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.621943    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.623655    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.625253    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.627021    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.628716    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.630406    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.631999    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.633700    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.635307    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.637038    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.638755    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.640456    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.642156    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.643899    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.645698    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.647813    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.649407    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.651187    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.653032    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.654881    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.656670    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.658477    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.660156    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.661835    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.663764    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.665554    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.667274    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.669210    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.671069    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.672763    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.674726    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.676490    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.678462    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.680314    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.682259    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.726345    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.727624    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.728962    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.730310    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.731669    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.733013    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.734349    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.735702    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.737031    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.738383    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.739749    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.741102    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.742441    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.743788    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.745122    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.746544    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.747930    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.749275    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.750657    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.752035    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.753365    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.754730    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.756078    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.757419    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.758771    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.760130    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.761499    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.762865    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.764232    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.765614    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.766965    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.768327    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.769693    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.771080    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.772490    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.773954    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.775471    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.776905    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.778317    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.779728    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.816947    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.818534    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.819902    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.821291    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.822660    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.824044    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.825474    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.826853    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.828220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.829603    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.830949    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.832327    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.833723    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.835114    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.836567    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.837948    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.839323    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.840680    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.842065    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.843465    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.844859    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.846248    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.847628    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.849001    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.850393    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.851763    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.853150    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.854616    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.856165    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.857561    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.858944    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.860338    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.861728    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.863182    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.864641    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.866035    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.867454    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.868907    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.898119    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.899538    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.900917    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.902276    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.903641    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.905116    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.906500    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.907924    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.909271    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.910649    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.912057    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.913437    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.914888    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.916324    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.917730    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.919245    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.920612    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.922012    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.923397    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.924825    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.926264    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.927671    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.929089    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.930500    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.931912    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.933352    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.934790    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.936194    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.937597    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.939018    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.940466    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.942197    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.943727    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.945391    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.947038    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.948511    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.950029    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.959793    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762273.972320    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.056325    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.057686    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.059119    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.060489    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.061855    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.063210    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.064586    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.065969    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.067302    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.068698    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.070106    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.071441    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.072798    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.074137    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.075520    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.076892    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.078296    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.079803    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.081147    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.082527    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.083943    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.085350    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.086777    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.088176    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.089626    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.091006    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.092434    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.093859    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.095250    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.096669    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.098068    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.099497    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.100874    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.102236    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.103634    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.105108    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.106611    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.108003    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.109406    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.110842    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.157538    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.159067    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.160647    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.162196    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.163784    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.165313    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.166873    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.168522    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.170168    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.171802    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.173468    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.175090    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.176753    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.178317    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.179855    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.181489    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.183316    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.185027    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.186689    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.188311    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.189956    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.191543    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.193168    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.194839    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.196442    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.198055    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.199665    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.201309    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.202907    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.204824    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.206535    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.208276    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.210197    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.212113    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.213915    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.215766    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.217381    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.256908    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.258204    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.259583    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.260943    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.262279    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.263642    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.265017    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.266384    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.267743    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.269094    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.270448    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.271815    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.273204    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.274573    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.275930    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.277325    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.278712    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.280133    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.281526    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.282929    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.284386    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.285789    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.287220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.288613    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.290019    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.291408    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.292782    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.294218    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.295615    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.297012    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.298398    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.299783    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.303277    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.304766    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.306267    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.307704    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.309127    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.310553    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.340111    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.341521    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.342869    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.344209    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.345535    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.346885    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.348241    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.349561    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.350904    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.352243    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.353606    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.354981    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.356335    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.357681    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.359027    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.360407    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.361827    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.363220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.364552    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.365940    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.367296    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.368732    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.370142    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.371505    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.372897    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.374283    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.375753    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.377131    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.378483    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.379827    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.381214    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.382638    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.384124    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.385521    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.386989    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.389133    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.390556    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.391967    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.393381    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.396167    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.435266    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.436655    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.438043    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.439416    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.440767    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.442115    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.443453    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.444851    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.446210    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.447567    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.448917    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.450270    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.451659    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.453064    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.454427    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.455794    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.457158    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.458592    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.459975    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.461402    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.462818    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.464211    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.465607    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.467009    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.468420    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.469858    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.471233    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.472636    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.474018    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.475394    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.476784    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.478198    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.479587    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.481067    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.482540    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.483957    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.485372    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.486781    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.488219    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.491695    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.520779    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.522083    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.523453    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.524792    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.526149    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.527504    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.528868    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.530208    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.531545    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.532901    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.534246    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.535613    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.536978    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.538315    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.539678    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.541082    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.542445    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.543856    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.545222    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.546614    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.548035    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.549441    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.550828    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.552218    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.553620    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.554983    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.556367    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.557767    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.559150    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.560560    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.561973    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.563339    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.564774    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.566254    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.567723    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.569134    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.570544    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.571963    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.607786    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.609107    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.610503    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.611869    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.613270    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.614627    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.616018    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.617380    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.618733    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.620104    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.621487    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.622941    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.624359    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.625726    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.627096    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.628453    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.629820    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.631193    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.632569    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.634030    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.635434    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.636842    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.638249    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.639668    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.641072    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.642461    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.643894    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.645312    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.646700    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.648108    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.649505    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.650905    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.652309    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.653830    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.655326    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.656748    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.658190    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.659642    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.698898    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.700212    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.701587    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.702975    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.704351    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.705716    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.707093    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.708455    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.709820    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.711243    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.712642    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.714016    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.715393    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.716766    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.718161    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.719537    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.720987    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.722363    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.723754    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.725144    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.726562    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.727983    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.729400    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.730836    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.732261    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.733659    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.735063    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.736471    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.737887    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.739309    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.740718    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.742127    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.743653    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.779408    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.780734    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.782091    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.783463    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.784823    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.786170    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.787550    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.788906    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.790256    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.791680    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.793078    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.794445    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.795810    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.797157    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.798523    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.799896    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.801343    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.802719    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.804094    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.805461    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.806875    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.808280    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.809685    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.811108    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.812521    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.813950    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.815391    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.816813    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.818220    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.819658    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.821081    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.822537    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.824067    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.851286    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.852593    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.853957    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.855310    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.856661    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.858005    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.859382    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.860726    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.862072    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.863442    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.864799    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.866218    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.867642    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.868998    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.870361    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.871724    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.873091    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.874469    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.875863    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.877310    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.878712    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.880116    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.881517    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.882924    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.884338    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.885710    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.887161    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.888576    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.890001    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.891408    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.892814    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.894239    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.895660    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.897197    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.898728    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.900159    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.901610    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.903074    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.967380    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.968906    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.970422    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.971879    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.973339    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.974775    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.976228    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.977673    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.979093    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.980559    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.981952    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.983349    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.984794    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.986199    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.987620    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.988987    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.990378    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.991780    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.993246    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.994729    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.996160    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.997593    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762274.999031    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.000500    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.001985    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.003406    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.004937    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.006397    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.007883    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.009332    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.010780    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.012282    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.013779    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.015359    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.016933    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.018391    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.019864    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.021359    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.059599    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.060901    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.062266    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.063666    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.065027    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.066373    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.067761    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.069105    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.070466    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.071868    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.073284    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.074660    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.076028    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.077428    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.078827    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.080203    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.081657    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.083036    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.084451    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.085898    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.087347    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.088765    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.090202    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.091736    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.093217    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.094619    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.096051    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.097473    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.098933    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.100413    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.101843    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.103294    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.104840    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.159623    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.161001    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.162389    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.163754    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.165104    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.166456    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.167833    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.169210    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.170574    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.171949    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.173353    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.174870    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.176268    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.177660    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.179031    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.180417    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.181861    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.183255    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.184671    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.186075    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.187514    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.188980    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.190503    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.191997    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.193473    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.194925    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.196403    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.197864    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.199331    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.200757    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.202218    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.203668    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.205125    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.206582    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.208054    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.209456    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.210904    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.212513    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.214134    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.215585    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.250477    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.251852    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.253254    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.254653    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.256020    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.257402    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.258802    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.260170    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.261551    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.262934    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.264387    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.265839    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.267254    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.268704    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.270085    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.271488    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.272910    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.274337    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.275761    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.277179    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.278678    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.280140    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.281659    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.283116    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.284589    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.286095    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.287613    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.289090    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.290564    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.292034    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.293551    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.294999    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.296485    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.297956    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.299468    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.300896    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.302321    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.303938    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.305555    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.307046    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.346895    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.348342    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.349873    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.351386    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.352872    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.354360    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.355938    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.357460    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.358996    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.360480    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.361978    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.363515    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.365166    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.366752    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.368324    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.369896    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.371475    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.373057    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.374581    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.376270    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.378022    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.379625    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.381335    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.382918    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.384579    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.386181    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.387898    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.389622    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.391400    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.393129    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.394848    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.396594    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.398270    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.399947    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.401512    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.403237    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.404878    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.406628    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.408659    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.456374    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.457691    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.459044    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.460450    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.461834    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.463211    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.464574    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.465925    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.467335    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.468748    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.470141    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.471520    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.472953    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.474398    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.475789    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.477174    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.478554    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.479966    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.481423    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.482850    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.484305    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.485725    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.487176    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.488664    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.490115    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.491543    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.492949    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.494358    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.495819    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.497229    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.498578    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.499926    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.501352    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.502817    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.504349    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.505886    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.507948    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.509417    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.510869    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.541079    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.542388    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.543789    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.545161    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.546531    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.547903    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.549289    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.550680    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.552064    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.553440    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.554844    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.556242    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.557639    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.559040    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.560411    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.561787    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.563175    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.564584    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.565984    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.567391    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.568820    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.570251    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.571716    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.573157    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.574578    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.575994    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.577397    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.578830    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.580256    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.581675    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.583019    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.584363    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.585858    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.587306    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
W0000 00:00:1729762275.588837    9183 gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced
Real Caption: <start> a kitchen with a lot of cabinet space and a tile wall <end>
Prediction Caption: some modern looking kitchen with an oven and a window <end>
No description has been provided for this image
No description has been provided for this image
************************************************************
Predicted Caption :
some modern looking kitchen with an oven and a window <end>

************************************************************
References :
<start> a room showing a kitchen with a microwave and a cooker <end>
<start> the lights are on in the kitchen with white cabinets <end>
<start> a small kitchen showing counter cabinets and sink <end>
<start> a kitchen with a lot of cabinet space and a tile wall <end>
<start> a look in to a very typical looking kitchen with white cabinets <end>
...

************************************************************
BLEU Score :
unigram  = 0.5454545455
bigram   = 0.4045199175
trigram  = 0.2664896292
4-gram = 0.0000000000
************************************************************
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 4-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
In [ ]:
# Optimizer and Checkpoint Management
checkpoint_path = "./checkpoints/lstm-incv3"

Model 6¶

LSTM and InceptionV3

In [ ]:
# Instantiate encoder and decoder
encoder_lstm_incv3, decoder_lstm_incv3 = create_model(CNN_Encoder, RNN_Decoder_LSTM, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_lstm_incv3,
                           decoder=decoder_lstm_incv3,
                           optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
                                          checkpoint_path,
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_lstm_incv3, decoder_lstm_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 1.9984
Epoch 1 Batch 100 Loss 1.1119
Epoch 1 Batch 200 Loss 1.0133
Epoch 1 Batch 300 Loss 0.9281
Epoch 1 Batch 400 Loss 0.8589
Epoch 1 Batch 500 Loss 0.7671
Epoch 1 Batch 600 Loss 0.8167
Epoch 1 Loss 0.991312
Time taken for 1 epoch 129.95868301391602 sec

Epoch 2 Batch 0 Loss 0.9500
Epoch 2 Batch 100 Loss 0.7699
Epoch 2 Batch 200 Loss 0.8934
Epoch 2 Batch 300 Loss 0.7875
Epoch 2 Batch 400 Loss 0.7850
Epoch 2 Batch 500 Loss 0.7493
Epoch 2 Batch 600 Loss 0.7019
Epoch 2 Loss 0.800343
Time taken for 1 epoch 77.09135317802429 sec

Epoch 3 Batch 0 Loss 0.7677
Epoch 3 Batch 100 Loss 0.7789
Epoch 3 Batch 200 Loss 0.7544
Epoch 3 Batch 300 Loss 0.7827
Epoch 3 Batch 400 Loss 0.7102
Epoch 3 Batch 500 Loss 0.7426
Epoch 3 Batch 600 Loss 0.7239
Epoch 3 Loss 0.727023
Time taken for 1 epoch 76.90206217765808 sec

Epoch 4 Batch 0 Loss 0.8144
Epoch 4 Batch 100 Loss 0.7971
Epoch 4 Batch 200 Loss 0.7578
Epoch 4 Batch 300 Loss 0.6326
Epoch 4 Batch 400 Loss 0.6539
Epoch 4 Batch 500 Loss 0.6167
Epoch 4 Batch 600 Loss 0.6240
Epoch 4 Loss 0.676870
Time taken for 1 epoch 77.05295157432556 sec

Epoch 5 Batch 0 Loss 0.5966
Epoch 5 Batch 100 Loss 0.6785
Epoch 5 Batch 200 Loss 0.7144
Epoch 5 Batch 300 Loss 0.6843
Epoch 5 Batch 400 Loss 0.6350
Epoch 5 Batch 500 Loss 0.5858
Epoch 5 Batch 600 Loss 0.5070
Epoch 5 Loss 0.635758
Time taken for 1 epoch 76.85066795349121 sec

Epoch 6 Batch 0 Loss 0.6737
Epoch 6 Batch 100 Loss 0.6642
Epoch 6 Batch 200 Loss 0.6621
Epoch 6 Batch 300 Loss 0.5639
Epoch 6 Batch 400 Loss 0.5286
Epoch 6 Batch 500 Loss 0.5816
Epoch 6 Batch 600 Loss 0.5438
Epoch 6 Loss 0.599297
Time taken for 1 epoch 77.16862893104553 sec

Epoch 7 Batch 0 Loss 0.6536
Epoch 7 Batch 100 Loss 0.6267
Epoch 7 Batch 200 Loss 0.4939
Epoch 7 Batch 300 Loss 0.5009
Epoch 7 Batch 400 Loss 0.5533
Epoch 7 Batch 500 Loss 0.5848
Epoch 7 Batch 600 Loss 0.5812
Epoch 7 Loss 0.565294
Time taken for 1 epoch 76.95766997337341 sec

Epoch 8 Batch 0 Loss 0.5668
Epoch 8 Batch 100 Loss 0.5441
Epoch 8 Batch 200 Loss 0.6014
Epoch 8 Batch 300 Loss 0.4699
Epoch 8 Batch 400 Loss 0.5002
Epoch 8 Batch 500 Loss 0.5269
Epoch 8 Batch 600 Loss 0.5465
Epoch 8 Loss 0.533389
Time taken for 1 epoch 77.0095465183258 sec

Epoch 9 Batch 0 Loss 0.5225
Epoch 9 Batch 100 Loss 0.5121
Epoch 9 Batch 200 Loss 0.5204
Epoch 9 Batch 300 Loss 0.5028
Epoch 9 Batch 400 Loss 0.5206
Epoch 9 Batch 500 Loss 0.4632
Epoch 9 Batch 600 Loss 0.5342
Epoch 9 Loss 0.503142
Time taken for 1 epoch 77.04063487052917 sec

Epoch 10 Batch 0 Loss 0.4582
Epoch 10 Batch 100 Loss 0.4926
Epoch 10 Batch 200 Loss 0.5259
Epoch 10 Batch 300 Loss 0.4841
Epoch 10 Batch 400 Loss 0.4431
Epoch 10 Batch 500 Loss 0.4098
Epoch 10 Batch 600 Loss 0.4590
Epoch 10 Loss 0.474277
Time taken for 1 epoch 76.95212960243225 sec

Epoch 11 Batch 0 Loss 0.4652
Epoch 11 Batch 100 Loss 0.4325
Epoch 11 Batch 200 Loss 0.4948
Epoch 11 Batch 300 Loss 0.4654
Epoch 11 Batch 400 Loss 0.4169
Epoch 11 Batch 500 Loss 0.3937
Epoch 11 Batch 600 Loss 0.4375
Epoch 11 Loss 0.446720
Time taken for 1 epoch 77.14709115028381 sec

Epoch 12 Batch 0 Loss 0.4276
Epoch 12 Batch 100 Loss 0.4503
Epoch 12 Batch 200 Loss 0.4158
Epoch 12 Batch 300 Loss 0.4238
Epoch 12 Batch 400 Loss 0.4349
Epoch 12 Batch 500 Loss 0.3745
Epoch 12 Batch 600 Loss 0.4211
Epoch 12 Loss 0.420779
Time taken for 1 epoch 76.95024394989014 sec

Epoch 13 Batch 0 Loss 0.4466
Epoch 13 Batch 100 Loss 0.3693
Epoch 13 Batch 200 Loss 0.4657
Epoch 13 Batch 300 Loss 0.4158
Epoch 13 Batch 400 Loss 0.3720
Epoch 13 Batch 500 Loss 0.3582
Epoch 13 Batch 600 Loss 0.3500
Epoch 13 Loss 0.397309
Time taken for 1 epoch 77.05328512191772 sec

Epoch 14 Batch 0 Loss 0.3655
Epoch 14 Batch 100 Loss 0.3873
Epoch 14 Batch 200 Loss 0.4350
Epoch 14 Batch 300 Loss 0.3454
Epoch 14 Batch 400 Loss 0.3851
Epoch 14 Batch 500 Loss 0.3204
Epoch 14 Batch 600 Loss 0.3528
Epoch 14 Loss 0.374772
Time taken for 1 epoch 77.13632464408875 sec

Epoch 15 Batch 0 Loss 0.3951
Epoch 15 Batch 100 Loss 0.3861
Epoch 15 Batch 200 Loss 0.4298
Epoch 15 Batch 300 Loss 0.3083
Epoch 15 Batch 400 Loss 0.3179
Epoch 15 Batch 500 Loss 0.3784
Epoch 15 Batch 600 Loss 0.3721
Epoch 15 Loss 0.355054
Time taken for 1 epoch 77.06284379959106 sec

Epoch 16 Batch 0 Loss 0.3485
Epoch 16 Batch 100 Loss 0.3445
Epoch 16 Batch 200 Loss 0.3333
Epoch 16 Batch 300 Loss 0.3346
Epoch 16 Batch 400 Loss 0.3403
Epoch 16 Batch 500 Loss 0.3445
Epoch 16 Batch 600 Loss 0.3397
Epoch 16 Loss 0.335110
Time taken for 1 epoch 77.70569515228271 sec

Epoch 17 Batch 0 Loss 0.3407
Epoch 17 Batch 100 Loss 0.3052
Epoch 17 Batch 200 Loss 0.3157
Epoch 17 Batch 300 Loss 0.3101
Epoch 17 Batch 400 Loss 0.3413
Epoch 17 Batch 500 Loss 0.3351
Epoch 17 Batch 600 Loss 0.2729
Epoch 17 Loss 0.317208
Time taken for 1 epoch 77.05879306793213 sec

Epoch 18 Batch 0 Loss 0.3027
Epoch 18 Batch 100 Loss 0.2742
Epoch 18 Batch 200 Loss 0.3092
Epoch 18 Batch 300 Loss 0.2876
Epoch 18 Batch 400 Loss 0.3139
Epoch 18 Batch 500 Loss 0.3329
Epoch 18 Batch 600 Loss 0.2722
Epoch 18 Loss 0.301593
Time taken for 1 epoch 76.95434212684631 sec

Epoch 19 Batch 0 Loss 0.3318
Epoch 19 Batch 100 Loss 0.2664
Epoch 19 Batch 200 Loss 0.2541
Epoch 19 Batch 300 Loss 0.2981
Epoch 19 Batch 400 Loss 0.2499
Epoch 19 Batch 500 Loss 0.2749
Epoch 19 Batch 600 Loss 0.2625
Epoch 19 Loss 0.285071
Time taken for 1 epoch 76.92547869682312 sec

Epoch 20 Batch 0 Loss 0.3262
Epoch 20 Batch 100 Loss 0.2832
Epoch 20 Batch 200 Loss 0.2784
Epoch 20 Batch 300 Loss 0.2378
Epoch 20 Batch 400 Loss 0.2854
Epoch 20 Batch 500 Loss 0.2705
Epoch 20 Batch 600 Loss 0.2510
Epoch 20 Loss 0.272564
Time taken for 1 epoch 77.00969767570496 sec

No description has been provided for this image
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder_lstm_incv3, decoder_lstm_incv3)

print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
/kaggle/working/train2014/COCO_train2014_000000199598.jpg
Real Caption: <start> this is a very large bathroom in an empty house <end>
Prediction Caption: a bathroom with a large mirror and a light on <end>
No description has been provided for this image
In [ ]:
display_bleu_score(image, result)
No description has been provided for this image
************************************************************
Predicted Caption :
a bathroom with a large mirror and a light on <end>

************************************************************
References :
<start> this an empty master bathroom of a vacant house <end>
<start> a bathroom with a large white tub next to a window <end>
<start> a bath tub sits in a large bathroom <end>
<start> a view of a large bathroom with a huge tub in it <end>
<start> this is a very large bathroom in an empty house <end>
...

************************************************************
BLEU Score :
unigram  = 0.6363636364
bigram   = 0.5045249791
trigram  = 0.4430498054
4-gram = 0.3816330911
************************************************************
In [ ]:
# Save the Encoder model
encoder_lstm_incv3.save('/kaggle/working/models/captioning_models/encoder_lstm_incv3.keras')

# Save the Decoder model
decoder_lstm_incv3.save('/kaggle/working/models/captioning_models/decoder_lstm_incv3.keras')
In [ ]:
evaluate_average_bleu(encoder_lstm_incv3, decoder_lstm_incv3, img_name_val, cap_val, tokenizer)
/opt/conda/lib/python3.10/site-packages/nltk/translate/bleu_score.py:490: UserWarning: 
Corpus/Sentence contains 0 counts of 2-gram overlaps.
BLEU scores might be undesirable; use SmoothingFunction().
  warnings.warn(_msg)
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.0914357750
bigram   = 0.2912023558
trigram  = 0.4352421055
4-gram = 0.5266112327
************************************************************
(0.09143577502154202,
 0.29120235581086157,
 0.4352421054856086,
 0.5266112326833678)
In [ ]:
encoder_gru_L3_incv3.save('models/captioning_models/encoder_IncV3_gru_model.keras')
decoder_gru_L3_incv3.save('models/captioning_models/decoder_IncV3_gru_model.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_L3_incv3, decoder_gru_L3_incv3, img_name_val, cap_val, tokenizer)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 2-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/home/arslane/Documents/CESI/DataScience/image_captioning_full_pipeline/.venv/lib/python3.10/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 3-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.1000155120
bigram   = 0.0000000000
trigram  = 0.0000000000
4-gram = 0.0000000000
************************************************************
(0.10001551200368086,
 4.456034186751795e-155,
 3.901438391495613e-204,
 9.59098435557862e-232)
In [ ]:
%system paplay /usr/share/sounds/freedesktop/stereo/complete.oga
[]

Model 7¶

LSTM with BLEU and InceptionV3

In [ ]:
# Optimizer and Checkpoint Management
checkpoint_path = "./checkpoints/lstm-incv3"
In [ ]:
# Instantiate encoder and decoder
encoder_lstm_incv3, decoder_lstm_incv3 = create_model(CNN_Encoder, RNN_Decoder_LSTM, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_lstm_incv3,
                           decoder=decoder_lstm_incv3,
                           optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
                                          checkpoint_path,
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_lstm_incv3, decoder_lstm_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 1.9984
Epoch 1 Batch 100 Loss 1.1119
Epoch 1 Batch 200 Loss 1.0133
Epoch 1 Batch 300 Loss 0.9281
Epoch 1 Batch 400 Loss 0.8589
Epoch 1 Batch 500 Loss 0.7671
Epoch 1 Batch 600 Loss 0.8167
Epoch 1 Loss 0.991312
Time taken for 1 epoch 129.95868301391602 sec

Epoch 2 Batch 0 Loss 0.9500
Epoch 2 Batch 100 Loss 0.7699
Epoch 2 Batch 200 Loss 0.8934
Epoch 2 Batch 300 Loss 0.7875
Epoch 2 Batch 400 Loss 0.7850
Epoch 2 Batch 500 Loss 0.7493
Epoch 2 Batch 600 Loss 0.7019
Epoch 2 Loss 0.800343
Time taken for 1 epoch 77.09135317802429 sec

Epoch 3 Batch 0 Loss 0.7677
Epoch 3 Batch 100 Loss 0.7789
Epoch 3 Batch 200 Loss 0.7544
Epoch 3 Batch 300 Loss 0.7827
Epoch 3 Batch 400 Loss 0.7102
Epoch 3 Batch 500 Loss 0.7426
Epoch 3 Batch 600 Loss 0.7239
Epoch 3 Loss 0.727023
Time taken for 1 epoch 76.90206217765808 sec

Epoch 4 Batch 0 Loss 0.8144
Epoch 4 Batch 100 Loss 0.7971
Epoch 4 Batch 200 Loss 0.7578
Epoch 4 Batch 300 Loss 0.6326
Epoch 4 Batch 400 Loss 0.6539
Epoch 4 Batch 500 Loss 0.6167
Epoch 4 Batch 600 Loss 0.6240
Epoch 4 Loss 0.676870
Time taken for 1 epoch 77.05295157432556 sec

Epoch 5 Batch 0 Loss 0.5966
Epoch 5 Batch 100 Loss 0.6785
Epoch 5 Batch 200 Loss 0.7144
Epoch 5 Batch 300 Loss 0.6843
Epoch 5 Batch 400 Loss 0.6350
Epoch 5 Batch 500 Loss 0.5858
Epoch 5 Batch 600 Loss 0.5070
Epoch 5 Loss 0.635758
Time taken for 1 epoch 76.85066795349121 sec

Epoch 6 Batch 0 Loss 0.6737
Epoch 6 Batch 100 Loss 0.6642
Epoch 6 Batch 200 Loss 0.6621
Epoch 6 Batch 300 Loss 0.5639
Epoch 6 Batch 400 Loss 0.5286
Epoch 6 Batch 500 Loss 0.5816
Epoch 6 Batch 600 Loss 0.5438
Epoch 6 Loss 0.599297
Time taken for 1 epoch 77.16862893104553 sec

Epoch 7 Batch 0 Loss 0.6536
Epoch 7 Batch 100 Loss 0.6267
Epoch 7 Batch 200 Loss 0.4939
Epoch 7 Batch 300 Loss 0.5009
Epoch 7 Batch 400 Loss 0.5533
Epoch 7 Batch 500 Loss 0.5848
Epoch 7 Batch 600 Loss 0.5812
Epoch 7 Loss 0.565294
Time taken for 1 epoch 76.95766997337341 sec

Epoch 8 Batch 0 Loss 0.5668
Epoch 8 Batch 100 Loss 0.5441
Epoch 8 Batch 200 Loss 0.6014
Epoch 8 Batch 300 Loss 0.4699
Epoch 8 Batch 400 Loss 0.5002
Epoch 8 Batch 500 Loss 0.5269
Epoch 8 Batch 600 Loss 0.5465
Epoch 8 Loss 0.533389
Time taken for 1 epoch 77.0095465183258 sec

Epoch 9 Batch 0 Loss 0.5225
Epoch 9 Batch 100 Loss 0.5121
Epoch 9 Batch 200 Loss 0.5204
Epoch 9 Batch 300 Loss 0.5028
Epoch 9 Batch 400 Loss 0.5206
Epoch 9 Batch 500 Loss 0.4632
Epoch 9 Batch 600 Loss 0.5342
Epoch 9 Loss 0.503142
Time taken for 1 epoch 77.04063487052917 sec

Epoch 10 Batch 0 Loss 0.4582
Epoch 10 Batch 100 Loss 0.4926
Epoch 10 Batch 200 Loss 0.5259
Epoch 10 Batch 300 Loss 0.4841
Epoch 10 Batch 400 Loss 0.4431
Epoch 10 Batch 500 Loss 0.4098
Epoch 10 Batch 600 Loss 0.4590
Epoch 10 Loss 0.474277
Time taken for 1 epoch 76.95212960243225 sec

Epoch 11 Batch 0 Loss 0.4652
Epoch 11 Batch 100 Loss 0.4325
Epoch 11 Batch 200 Loss 0.4948
Epoch 11 Batch 300 Loss 0.4654
Epoch 11 Batch 400 Loss 0.4169
Epoch 11 Batch 500 Loss 0.3937
Epoch 11 Batch 600 Loss 0.4375
Epoch 11 Loss 0.446720
Time taken for 1 epoch 77.14709115028381 sec

Epoch 12 Batch 0 Loss 0.4276
Epoch 12 Batch 100 Loss 0.4503
Epoch 12 Batch 200 Loss 0.4158
Epoch 12 Batch 300 Loss 0.4238
Epoch 12 Batch 400 Loss 0.4349
Epoch 12 Batch 500 Loss 0.3745
Epoch 12 Batch 600 Loss 0.4211
Epoch 12 Loss 0.420779
Time taken for 1 epoch 76.95024394989014 sec

Epoch 13 Batch 0 Loss 0.4466
Epoch 13 Batch 100 Loss 0.3693
Epoch 13 Batch 200 Loss 0.4657
Epoch 13 Batch 300 Loss 0.4158
Epoch 13 Batch 400 Loss 0.3720
Epoch 13 Batch 500 Loss 0.3582
Epoch 13 Batch 600 Loss 0.3500
Epoch 13 Loss 0.397309
Time taken for 1 epoch 77.05328512191772 sec

Epoch 14 Batch 0 Loss 0.3655
Epoch 14 Batch 100 Loss 0.3873
Epoch 14 Batch 200 Loss 0.4350
Epoch 14 Batch 300 Loss 0.3454
Epoch 14 Batch 400 Loss 0.3851
Epoch 14 Batch 500 Loss 0.3204
Epoch 14 Batch 600 Loss 0.3528
Epoch 14 Loss 0.374772
Time taken for 1 epoch 77.13632464408875 sec

Epoch 15 Batch 0 Loss 0.3951
Epoch 15 Batch 100 Loss 0.3861
Epoch 15 Batch 200 Loss 0.4298
Epoch 15 Batch 300 Loss 0.3083
Epoch 15 Batch 400 Loss 0.3179
Epoch 15 Batch 500 Loss 0.3784
Epoch 15 Batch 600 Loss 0.3721
Epoch 15 Loss 0.355054
Time taken for 1 epoch 77.06284379959106 sec

Epoch 16 Batch 0 Loss 0.3485
Epoch 16 Batch 100 Loss 0.3445
Epoch 16 Batch 200 Loss 0.3333
Epoch 16 Batch 300 Loss 0.3346
Epoch 16 Batch 400 Loss 0.3403
Epoch 16 Batch 500 Loss 0.3445
Epoch 16 Batch 600 Loss 0.3397
Epoch 16 Loss 0.335110
Time taken for 1 epoch 77.70569515228271 sec

Epoch 17 Batch 0 Loss 0.3407
Epoch 17 Batch 100 Loss 0.3052
Epoch 17 Batch 200 Loss 0.3157
Epoch 17 Batch 300 Loss 0.3101
Epoch 17 Batch 400 Loss 0.3413
Epoch 17 Batch 500 Loss 0.3351
Epoch 17 Batch 600 Loss 0.2729
Epoch 17 Loss 0.317208
Time taken for 1 epoch 77.05879306793213 sec

Epoch 18 Batch 0 Loss 0.3027
Epoch 18 Batch 100 Loss 0.2742
Epoch 18 Batch 200 Loss 0.3092
Epoch 18 Batch 300 Loss 0.2876
Epoch 18 Batch 400 Loss 0.3139
Epoch 18 Batch 500 Loss 0.3329
Epoch 18 Batch 600 Loss 0.2722
Epoch 18 Loss 0.301593
Time taken for 1 epoch 76.95434212684631 sec

Epoch 19 Batch 0 Loss 0.3318
Epoch 19 Batch 100 Loss 0.2664
Epoch 19 Batch 200 Loss 0.2541
Epoch 19 Batch 300 Loss 0.2981
Epoch 19 Batch 400 Loss 0.2499
Epoch 19 Batch 500 Loss 0.2749
Epoch 19 Batch 600 Loss 0.2625
Epoch 19 Loss 0.285071
Time taken for 1 epoch 76.92547869682312 sec

Epoch 20 Batch 0 Loss 0.3262
Epoch 20 Batch 100 Loss 0.2832
Epoch 20 Batch 200 Loss 0.2784
Epoch 20 Batch 300 Loss 0.2378
Epoch 20 Batch 400 Loss 0.2854
Epoch 20 Batch 500 Loss 0.2705
Epoch 20 Batch 600 Loss 0.2510
Epoch 20 Loss 0.272564
Time taken for 1 epoch 77.00969767570496 sec

No description has been provided for this image
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder_lstm_incv3, decoder_lstm_incv3)

print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
/kaggle/working/train2014/COCO_train2014_000000199598.jpg
Real Caption: <start> this is a very large bathroom in an empty house <end>
Prediction Caption: a bathroom with a large mirror and a light on <end>
No description has been provided for this image
In [ ]:
display_bleu_score(image, result)
No description has been provided for this image
************************************************************
Predicted Caption :
a bathroom with a large mirror and a light on <end>

************************************************************
References :
<start> this an empty master bathroom of a vacant house <end>
<start> a bathroom with a large white tub next to a window <end>
<start> a bath tub sits in a large bathroom <end>
<start> a view of a large bathroom with a huge tub in it <end>
<start> this is a very large bathroom in an empty house <end>
...

************************************************************
BLEU Score :
unigram  = 0.6363636364
bigram   = 0.5045249791
trigram  = 0.4430498054
4-gram = 0.3816330911
************************************************************
In [ ]:
# Save the Encoder model
encoder_lstm_incv3.save('/kaggle/working/models/captioning_models/encoder_lstm_incv3.keras')

# Save the Decoder model
decoder_lstm_incv3.save('/kaggle/working/models/captioning_models/decoder_lstm_incv3.keras')
In [ ]:
evaluate_average_bleu(encoder_lstm_incv3, decoder_lstm_incv3, img_name_val, cap_val, tokenizer)
/opt/conda/lib/python3.10/site-packages/nltk/translate/bleu_score.py:490: UserWarning: 
Corpus/Sentence contains 0 counts of 2-gram overlaps.
BLEU scores might be undesirable; use SmoothingFunction().
  warnings.warn(_msg)
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.0914357750
bigram   = 0.2912023558
trigram  = 0.4352421055
4-gram = 0.5266112327
************************************************************
(0.09143577502154202,
 0.29120235581086157,
 0.4352421054856086,
 0.5266112326833678)

Model 7¶

GRU with BLEU and InceptionV3

In [ ]:
# Instantiate encoder and decoder
encoder_gru_incv3, decoder_gru_incv3 = create_model(CNN_Encoder, RNN_Decoder_GRU, embedding_dim, units, vocab_size)

# Checkpoint setup
ckpt = tf.train.Checkpoint(encoder=encoder_gru_incv3,
                           decoder=decoder_gru_incv3,
                           optimizer=optimizer)
ckpt_manager = tf.train.CheckpointManager(ckpt,
                                          checkpoint_path,
                                          max_to_keep=5)

# Resume training from last checkpoint if exists
start_epoch = 0
if ckpt_manager.latest_checkpoint:
    start_epoch = int(ckpt_manager.latest_checkpoint.split('-')[-1])
    ckpt.restore(ckpt_manager.latest_checkpoint)
In [ ]:
train_model(encoder_gru_incv3, decoder_gru_incv3, ckpt_manager)
Epoch 1 Batch 0 Loss 1.8949
Epoch 1 Batch 100 Loss 0.9383
Epoch 1 Batch 200 Loss 0.7920
Epoch 1 Batch 300 Loss 0.8357
Epoch 1 Batch 400 Loss 0.7962
Epoch 1 Batch 500 Loss 0.7882
Epoch 1 Batch 600 Loss 0.6864
Epoch 1 Loss 0.850178
Time taken for 1 epoch 136.3702175617218 sec

Epoch 2 Batch 0 Loss 0.8000
Epoch 2 Batch 100 Loss 0.7817
Epoch 2 Batch 200 Loss 0.7412
Epoch 2 Batch 300 Loss 0.7135
Epoch 2 Batch 400 Loss 0.6566
Epoch 2 Batch 500 Loss 0.7199
Epoch 2 Batch 600 Loss 0.6245
Epoch 2 Loss 0.715335
Time taken for 1 epoch 67.74349689483643 sec

Epoch 3 Batch 0 Loss 0.6831
Epoch 3 Batch 100 Loss 0.6933
Epoch 3 Batch 200 Loss 0.6508
Epoch 3 Batch 300 Loss 0.6507
Epoch 3 Batch 400 Loss 0.6452
Epoch 3 Batch 500 Loss 0.6322
Epoch 3 Batch 600 Loss 0.6261
Epoch 3 Loss 0.661079
Time taken for 1 epoch 67.64078164100647 sec

Epoch 4 Batch 0 Loss 0.7098
Epoch 4 Batch 100 Loss 0.5877
Epoch 4 Batch 200 Loss 0.5796
Epoch 4 Batch 300 Loss 0.5457
Epoch 4 Batch 400 Loss 0.6382
Epoch 4 Batch 500 Loss 0.5834
Epoch 4 Batch 600 Loss 0.5630
Epoch 4 Loss 0.627894
Time taken for 1 epoch 67.46794772148132 sec

Epoch 5 Batch 0 Loss 0.6474
Epoch 5 Batch 100 Loss 0.5899
Epoch 5 Batch 200 Loss 0.6607
Epoch 5 Batch 300 Loss 0.5589
Epoch 5 Batch 400 Loss 0.6181
Epoch 5 Batch 500 Loss 0.5543
Epoch 5 Batch 600 Loss 0.6226
Epoch 5 Loss 0.602535
Time taken for 1 epoch 67.5987856388092 sec

Epoch 6 Batch 0 Loss 0.6641
Epoch 6 Batch 100 Loss 0.5438
Epoch 6 Batch 200 Loss 0.5980
Epoch 6 Batch 300 Loss 0.6187
Epoch 6 Batch 400 Loss 0.6684
Epoch 6 Batch 500 Loss 0.5840
Epoch 6 Batch 600 Loss 0.6016
Epoch 6 Loss 0.581196
Time taken for 1 epoch 67.73905658721924 sec

Epoch 7 Batch 0 Loss 0.6164
Epoch 7 Batch 100 Loss 0.5608
Epoch 7 Batch 200 Loss 0.5554
Epoch 7 Batch 300 Loss 0.5353
Epoch 7 Batch 400 Loss 0.5888
Epoch 7 Batch 500 Loss 0.5284
Epoch 7 Batch 600 Loss 0.5147
Epoch 7 Loss 0.562486
Time taken for 1 epoch 67.7080078125 sec

Epoch 8 Batch 0 Loss 0.4977
Epoch 8 Batch 100 Loss 0.5599
Epoch 8 Batch 200 Loss 0.5521
Epoch 8 Batch 300 Loss 0.5285
Epoch 8 Batch 400 Loss 0.5313
Epoch 8 Batch 500 Loss 0.5224
Epoch 8 Batch 600 Loss 0.5081
Epoch 8 Loss 0.546036
Time taken for 1 epoch 67.9389157295227 sec

Epoch 9 Batch 0 Loss 0.5639
Epoch 9 Batch 100 Loss 0.5185
Epoch 9 Batch 200 Loss 0.5130
Epoch 9 Batch 300 Loss 0.4707
Epoch 9 Batch 400 Loss 0.5226
Epoch 9 Batch 500 Loss 0.5308
Epoch 9 Batch 600 Loss 0.5287
Epoch 9 Loss 0.529052
Time taken for 1 epoch 68.5266101360321 sec

Epoch 10 Batch 0 Loss 0.5091
Epoch 10 Batch 100 Loss 0.5489
Epoch 10 Batch 200 Loss 0.4935
Epoch 10 Batch 300 Loss 0.4904
Epoch 10 Batch 400 Loss 0.5132
Epoch 10 Batch 500 Loss 0.5473
Epoch 10 Batch 600 Loss 0.5261
Epoch 10 Loss 0.513569
Time taken for 1 epoch 68.12801694869995 sec

Epoch 11 Batch 0 Loss 0.5286
Epoch 11 Batch 100 Loss 0.4907
Epoch 11 Batch 200 Loss 0.4971
Epoch 11 Batch 300 Loss 0.4784
Epoch 11 Batch 400 Loss 0.4716
Epoch 11 Batch 500 Loss 0.5123
Epoch 11 Batch 600 Loss 0.5165
Epoch 11 Loss 0.498211
Time taken for 1 epoch 68.70136260986328 sec

Epoch 12 Batch 0 Loss 0.4602
Epoch 12 Batch 100 Loss 0.4843
Epoch 12 Batch 200 Loss 0.4787
Epoch 12 Batch 300 Loss 0.4907
Epoch 12 Batch 400 Loss 0.4543
Epoch 12 Batch 500 Loss 0.5139
Epoch 12 Batch 600 Loss 0.4508
Epoch 12 Loss 0.483153
Time taken for 1 epoch 68.926766872406 sec

Epoch 13 Batch 0 Loss 0.5246
Epoch 13 Batch 100 Loss 0.4433
Epoch 13 Batch 200 Loss 0.4333
Epoch 13 Batch 300 Loss 0.5092
Epoch 13 Batch 400 Loss 0.5386
Epoch 13 Batch 500 Loss 0.4651
Epoch 13 Batch 600 Loss 0.5008
Epoch 13 Loss 0.467838
Time taken for 1 epoch 68.38380694389343 sec

Epoch 14 Batch 0 Loss 0.4670
Epoch 14 Batch 100 Loss 0.4697
Epoch 14 Batch 200 Loss 0.4244
Epoch 14 Batch 300 Loss 0.4041
Epoch 14 Batch 400 Loss 0.4178
Epoch 14 Batch 500 Loss 0.4842
Epoch 14 Batch 600 Loss 0.4065
Epoch 14 Loss 0.453543
Time taken for 1 epoch 67.91882824897766 sec

Epoch 15 Batch 0 Loss 0.4450
Epoch 15 Batch 100 Loss 0.4496
Epoch 15 Batch 200 Loss 0.4370
Epoch 15 Batch 300 Loss 0.3641
Epoch 15 Batch 400 Loss 0.4662
Epoch 15 Batch 500 Loss 0.4919
Epoch 15 Batch 600 Loss 0.3833
Epoch 15 Loss 0.440085
Time taken for 1 epoch 67.74384880065918 sec

Epoch 16 Batch 0 Loss 0.4519
Epoch 16 Batch 100 Loss 0.3983
Epoch 16 Batch 200 Loss 0.3945
Epoch 16 Batch 300 Loss 0.4124
Epoch 16 Batch 400 Loss 0.4057
Epoch 16 Batch 500 Loss 0.4294
Epoch 16 Batch 600 Loss 0.4456
Epoch 16 Loss 0.425590
Time taken for 1 epoch 68.48339438438416 sec

Epoch 17 Batch 0 Loss 0.3461
Epoch 17 Batch 100 Loss 0.3895
Epoch 17 Batch 200 Loss 0.3992
Epoch 17 Batch 300 Loss 0.4126
Epoch 17 Batch 400 Loss 0.3693
Epoch 17 Batch 500 Loss 0.4303
Epoch 17 Batch 600 Loss 0.3836
Epoch 17 Loss 0.414078
Time taken for 1 epoch 68.02987790107727 sec

Epoch 18 Batch 0 Loss 0.4320
Epoch 18 Batch 100 Loss 0.3899
Epoch 18 Batch 200 Loss 0.4164
Epoch 18 Batch 300 Loss 0.3417
Epoch 18 Batch 400 Loss 0.4256
Epoch 18 Batch 500 Loss 0.3774
Epoch 18 Batch 600 Loss 0.4188
Epoch 18 Loss 0.399674
Time taken for 1 epoch 67.78015422821045 sec

Epoch 19 Batch 0 Loss 0.4493
Epoch 19 Batch 100 Loss 0.3805
Epoch 19 Batch 200 Loss 0.4050
Epoch 19 Batch 300 Loss 0.4056
Epoch 19 Batch 400 Loss 0.4120
Epoch 19 Batch 500 Loss 0.3622
Epoch 19 Batch 600 Loss 0.3953
Epoch 19 Loss 0.387221
Time taken for 1 epoch 67.89082050323486 sec

Epoch 20 Batch 0 Loss 0.3754
Epoch 20 Batch 100 Loss 0.3667
Epoch 20 Batch 200 Loss 0.3888
Epoch 20 Batch 300 Loss 0.4120
Epoch 20 Batch 400 Loss 0.3742
Epoch 20 Batch 500 Loss 0.3683
Epoch 20 Batch 600 Loss 0.4101
Epoch 20 Loss 0.374367
Time taken for 1 epoch 68.14571762084961 sec

No description has been provided for this image
In [ ]:
# Affichage de quelques annotations dans le jeu de test
rid = np.random.randint(0, len(img_name_val))
image = img_name_val[rid]
print(image)
real_caption = ' '.join([tokenizer.index_word[i] for i in cap_val[rid] if i not in [0]])

result, attention_plot = evaluate(image, encoder_gru_incv3, decoder_gru_incv3)

print ('Real Caption:', real_caption)
print ('Prediction Caption:', ' '.join(result))
plot_attention(image, result, attention_plot)
/kaggle/working/train2014/COCO_train2014_000000501059.jpg
Real Caption: <start> a bunch of cars parked on the side of the road <end>
Prediction Caption: a car is parked cars <end>
No description has been provided for this image
In [ ]:
display_bleu_score(image, result)
No description has been provided for this image
************************************************************
Predicted Caption :
a car is parked cars <end>

************************************************************
References :
<start> a parking meter on a road with cars parked nearby <end>
<start> a bunch of cars parked on the side of the road <end>
<start> a view of street from curb with cars parked on side of street rows of parking meters trash bin with trees <end>
<start> a street filled with parked cars next to parking meters <end>
<start> several cars parked on the side of the road <end>
...

************************************************************
BLEU Score :
unigram  = 0.2897321390
bigram   = 0.1586928282
trigram  = 0.2235206825
4-gram = 0.2626168670
************************************************************
In [ ]:
# Save the Encoder model
encoder_gru_incv3.save('models/captioning_models/encoder_gru_incv3.keras')

# Save the Decoder model
decoder_gru_incv3.save('models/captioning_models/decoder_gru_incv3.keras')
In [ ]:
evaluate_average_bleu(encoder_gru_incv3, decoder_gru_incv3, img_name_val, cap_val, tokenizer)
Processed 0/5003 images
Processed 500/5003 images
Processed 1000/5003 images
Processed 1500/5003 images
Processed 2000/5003 images
Processed 2500/5003 images
Processed 3000/5003 images
Processed 3500/5003 images
Processed 4000/5003 images
Processed 4500/5003 images
Processed 5000/5003 images

************************************************************
Average BLEU Score on Validation Dataset:
unigram  = 0.0912703781
bigram   = 0.2830066358
trigram  = 0.4199234109
4-gram = 0.5064984794
************************************************************
(0.0912703780917766, 0.28300663576604557, 0.41992341086909, 0.5064984794036137)